Jul 29 18:32:12.761273 ip-10-0-137-210 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 29 18:32:12.761283 ip-10-0-137-210 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 29 18:32:12.761292 ip-10-0-137-210 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 29 18:32:12.761595 ip-10-0-137-210 systemd[1]: Failed to start Kubernetes Kubelet. Jul 29 18:32:22.960983 ip-10-0-137-210 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 29 18:32:22.961003 ip-10-0-137-210 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 6634912079c244259f5eb2b5266410bb -- Jul 29 18:34:20.184228 ip-10-0-137-210 systemd[1]: Starting Kubernetes Kubelet... Jul 29 18:34:20.517138 ip-10-0-137-210 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:20.517138 ip-10-0-137-210 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 29 18:34:20.517138 ip-10-0-137-210 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:20.517138 ip-10-0-137-210 kubenswrapper[2574]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 29 18:34:20.517138 ip-10-0-137-210 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 29 18:34:20.519300 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.519204 2574 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 29 18:34:20.525022 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525003 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525024 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525029 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525041 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525045 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525049 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525051 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525055 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525057 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525061 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525064 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525067 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525071 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525074 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525077 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:20.525072 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525080 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525083 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525086 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525088 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525091 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525094 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525097 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525100 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525102 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525105 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525107 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525110 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525113 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525115 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525118 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525120 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525123 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525126 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525128 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525130 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:20.525426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525133 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525135 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525138 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525140 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525143 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525145 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525148 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525150 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525158 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525161 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525163 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525166 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525168 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525171 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525174 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525177 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525180 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525182 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525185 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525187 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:20.525898 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525190 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525192 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525195 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525197 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525200 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525203 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525205 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525208 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525211 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525213 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525216 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525218 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525221 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525225 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525229 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525231 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525235 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525239 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525242 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:20.526412 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525245 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525247 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525255 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525258 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525260 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525263 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525265 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525269 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525272 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525274 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525277 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525280 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525282 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525285 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525287 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525290 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525292 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525295 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525298 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:20.526871 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525300 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525302 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525305 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525307 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525310 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.525313 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526085 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526092 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526098 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526102 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526105 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526108 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526110 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526113 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526116 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526118 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526121 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526124 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526127 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526129 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:20.527350 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526132 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526135 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526138 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526141 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526144 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526147 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526150 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526152 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526155 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526158 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526160 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526163 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526165 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526168 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526171 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526173 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526176 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526179 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526181 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:20.527972 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526184 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526186 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526189 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526191 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526194 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526196 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526198 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526201 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526203 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526206 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526209 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526212 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526215 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526217 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526220 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526223 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526225 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526228 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526231 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526233 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:20.528439 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526236 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526238 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526241 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526243 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526246 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526248 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526251 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526253 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526256 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526260 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526263 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526265 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526268 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526272 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526276 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526279 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526281 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526284 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526286 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:20.528934 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526290 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526293 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526296 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526299 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526301 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526304 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526308 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526311 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526314 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526317 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526320 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526322 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526325 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526327 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526330 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526333 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526335 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526338 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526340 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526342 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:20.529426 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526345 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526348 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526350 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526353 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526355 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526358 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526360 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526815 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526820 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526824 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526827 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526830 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526833 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526836 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526839 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526842 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526845 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526849 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526852 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526855 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:20.529912 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526858 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526860 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526863 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526865 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526868 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526871 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526873 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526876 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526879 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526881 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526884 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526886 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526889 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526892 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526895 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526897 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526900 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526902 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526904 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:20.530436 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526907 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526909 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526912 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526930 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526933 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526935 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526938 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526940 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526943 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526945 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526949 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526951 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526954 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526957 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526959 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526962 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526965 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526967 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526970 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526973 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:20.530951 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526975 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526977 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526980 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526984 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526988 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526991 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526995 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.526999 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527002 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527005 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527007 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527010 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527013 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527016 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527019 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527021 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527024 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527028 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:20.531430 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527031 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527035 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527037 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527040 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527043 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527047 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527050 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527052 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527055 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527058 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527060 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527063 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527067 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527069 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527072 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527074 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527077 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527079 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527082 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527084 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:20.531872 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527087 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527089 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527092 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527094 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527097 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527100 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527102 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527105 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527107 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527188 2574 flags.go:64] FLAG: --address="0.0.0.0" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527200 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527210 2574 flags.go:64] FLAG: --anonymous-auth="true" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527217 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527223 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527226 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527230 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527234 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527238 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527241 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527245 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527248 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527251 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 29 18:34:20.532402 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527253 2574 flags.go:64] FLAG: --cgroup-root="" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527256 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527259 2574 flags.go:64] FLAG: --client-ca-file="" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527262 2574 flags.go:64] FLAG: --cloud-provider="external" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527265 2574 flags.go:64] FLAG: --cluster-dns="[]" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527269 2574 flags.go:64] FLAG: --cluster-domain="" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527271 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527275 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527278 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527281 2574 flags.go:64] FLAG: --container-log-max-files="5" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527284 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527287 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527290 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527293 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527296 2574 flags.go:64] FLAG: --contention-profiling="false" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527299 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527301 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527305 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527308 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527312 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527315 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527317 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527320 2574 flags.go:64] FLAG: --enable-load-reader="false" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527323 2574 flags.go:64] FLAG: --enable-server="true" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527326 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 29 18:34:20.532942 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527331 2574 flags.go:64] FLAG: --event-burst="100" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527334 2574 flags.go:64] FLAG: --event-qps="50" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527343 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527349 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527354 2574 flags.go:64] FLAG: --eviction-hard="" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527358 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527361 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527364 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527367 2574 flags.go:64] FLAG: --eviction-soft="" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527370 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527372 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527375 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527378 2574 flags.go:64] FLAG: --experimental-mounter-path="" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527381 2574 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527384 2574 flags.go:64] FLAG: --fail-swap-on="true" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527387 2574 flags.go:64] FLAG: --feature-gates="" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527390 2574 flags.go:64] FLAG: --file-check-frequency="20s" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527393 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527397 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527400 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527403 2574 flags.go:64] FLAG: --healthz-port="10248" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527406 2574 flags.go:64] FLAG: --help="false" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527409 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527412 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 29 18:34:20.533544 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527414 2574 flags.go:64] FLAG: --http-check-frequency="20s" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527417 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527423 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527426 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527429 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527432 2574 flags.go:64] FLAG: --image-service-endpoint="" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527434 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527437 2574 flags.go:64] FLAG: --kube-api-burst="100" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527440 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527443 2574 flags.go:64] FLAG: --kube-api-qps="50" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527448 2574 flags.go:64] FLAG: --kube-reserved="" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527451 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527454 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527457 2574 flags.go:64] FLAG: --kubelet-cgroups="" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527460 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527463 2574 flags.go:64] FLAG: --lock-file="" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527466 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527468 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527471 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527477 2574 flags.go:64] FLAG: --log-json-split-stream="false" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527480 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527483 2574 flags.go:64] FLAG: --log-text-split-stream="false" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527486 2574 flags.go:64] FLAG: --logging-format="text" Jul 29 18:34:20.534183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527488 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527492 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527494 2574 flags.go:64] FLAG: --manifest-url="" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527497 2574 flags.go:64] FLAG: --manifest-url-header="" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527501 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527504 2574 flags.go:64] FLAG: --max-open-files="1000000" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527508 2574 flags.go:64] FLAG: --max-pods="110" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527511 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527514 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527517 2574 flags.go:64] FLAG: --memory-manager-policy="None" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527520 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527524 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527527 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527530 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527538 2574 flags.go:64] FLAG: --node-status-max-images="50" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527542 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527544 2574 flags.go:64] FLAG: --oom-score-adj="-999" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527548 2574 flags.go:64] FLAG: --pod-cidr="" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527551 2574 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527556 2574 flags.go:64] FLAG: --pod-manifest-path="" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527562 2574 flags.go:64] FLAG: --pod-max-pids="-1" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527565 2574 flags.go:64] FLAG: --pods-per-core="0" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527568 2574 flags.go:64] FLAG: --port="10250" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527571 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 29 18:34:20.534804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527574 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0996d6d70d875afa1" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527577 2574 flags.go:64] FLAG: --qos-reserved="" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527580 2574 flags.go:64] FLAG: --read-only-port="10255" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527583 2574 flags.go:64] FLAG: --register-node="true" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527586 2574 flags.go:64] FLAG: --register-with-taints="" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527589 2574 flags.go:64] FLAG: --registry-burst="10" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527592 2574 flags.go:64] FLAG: --registry-qps="5" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527595 2574 flags.go:64] FLAG: --reserved-cpus="" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527597 2574 flags.go:64] FLAG: --reserved-memory="" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527601 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527604 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527606 2574 flags.go:64] FLAG: --rotate-certificates="false" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527609 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527612 2574 flags.go:64] FLAG: --runonce="false" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527614 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527617 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527620 2574 flags.go:64] FLAG: --seccomp-default="false" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527623 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527626 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527630 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527633 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527636 2574 flags.go:64] FLAG: --storage-driver-password="root" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527639 2574 flags.go:64] FLAG: --storage-driver-secure="false" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527642 2574 flags.go:64] FLAG: --storage-driver-table="stats" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527644 2574 flags.go:64] FLAG: --storage-driver-user="root" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527647 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 29 18:34:20.535766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527650 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527653 2574 flags.go:64] FLAG: --system-cgroups="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527657 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527662 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527665 2574 flags.go:64] FLAG: --tls-cert-file="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527668 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527672 2574 flags.go:64] FLAG: --tls-min-version="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527675 2574 flags.go:64] FLAG: --tls-private-key-file="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527678 2574 flags.go:64] FLAG: --topology-manager-policy="none" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527681 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527683 2574 flags.go:64] FLAG: --topology-manager-scope="container" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527686 2574 flags.go:64] FLAG: --v="2" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527690 2574 flags.go:64] FLAG: --version="false" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527694 2574 flags.go:64] FLAG: --vmodule="" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527698 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.527701 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527845 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527850 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527853 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527857 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527860 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527864 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527867 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:20.536565 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527870 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527872 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527876 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527879 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527882 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527884 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527887 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527890 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527893 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527895 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527898 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527902 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527904 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527907 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527911 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527928 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527938 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527941 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527943 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527946 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:20.537171 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527949 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527951 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527954 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527957 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527959 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527962 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527964 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527967 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527969 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527972 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527975 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527977 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527980 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527983 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527987 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527989 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527992 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527994 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.527997 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528000 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:20.537657 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528002 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528005 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528007 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528011 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528014 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528016 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528019 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528021 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528024 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528032 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528035 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528037 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528040 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528043 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528045 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528048 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528050 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528053 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528055 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528058 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:20.538168 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528060 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528063 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528065 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528068 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528070 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528073 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528076 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528079 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528083 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528086 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528090 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528092 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528095 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528098 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528100 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528105 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528107 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528110 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528113 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:20.538651 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528115 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528118 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528121 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528129 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528131 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528134 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528136 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528140 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528144 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528147 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528149 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528152 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.528155 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.529024 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.537773 2574 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 29 18:34:20.539129 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.537790 2574 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.537817 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.537827 2574 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537897 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537904 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537909 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537932 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537937 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537940 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537943 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537946 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537949 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537951 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537954 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537957 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537959 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537962 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537965 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:20.539496 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537968 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537970 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537973 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537975 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537978 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537981 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537984 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537987 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537989 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537992 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537994 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.537997 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538000 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538002 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538006 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538008 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538011 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538014 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538017 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538019 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:20.540002 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538022 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538024 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538027 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538029 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538032 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538034 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538038 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538042 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538045 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538048 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538050 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538054 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538056 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538059 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538061 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538064 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538066 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538069 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538071 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538074 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:20.540476 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538076 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538079 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538082 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538085 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538087 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538090 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538093 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538096 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538099 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538101 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538104 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538106 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538109 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538111 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538114 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538117 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538119 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538121 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538124 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538127 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:20.541082 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538129 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538131 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538134 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538136 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538140 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538142 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538145 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538148 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538150 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538153 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538156 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538159 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538161 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538164 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538167 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538170 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538173 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538175 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538178 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538180 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:20.541566 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538183 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538186 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538189 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538191 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.538197 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538349 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538355 2574 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538358 2574 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538361 2574 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538364 2574 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538367 2574 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538370 2574 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538373 2574 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538376 2574 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538379 2574 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 29 18:34:20.542062 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538381 2574 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538385 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538388 2574 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538391 2574 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538394 2574 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538396 2574 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538400 2574 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538404 2574 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538407 2574 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538410 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538413 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538416 2574 feature_gate.go:328] unrecognized feature gate: Example2 Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538419 2574 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538421 2574 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538424 2574 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538426 2574 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538429 2574 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538431 2574 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538434 2574 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 29 18:34:20.542424 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538437 2574 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538441 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538444 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538446 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538449 2574 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538451 2574 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538454 2574 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538457 2574 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538459 2574 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538462 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538465 2574 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538467 2574 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538470 2574 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538472 2574 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538475 2574 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538477 2574 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538480 2574 feature_gate.go:328] unrecognized feature gate: Example Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538483 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538486 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538488 2574 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 29 18:34:20.542870 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538491 2574 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538493 2574 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538496 2574 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538498 2574 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538501 2574 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538503 2574 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538506 2574 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538509 2574 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538511 2574 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538514 2574 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538516 2574 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538519 2574 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538521 2574 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538524 2574 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538526 2574 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538529 2574 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538532 2574 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538534 2574 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538537 2574 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538539 2574 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 29 18:34:20.543368 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538541 2574 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538544 2574 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538546 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538549 2574 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538551 2574 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538554 2574 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538556 2574 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538559 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538561 2574 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538564 2574 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538567 2574 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538570 2574 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538572 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538574 2574 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538577 2574 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538580 2574 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538582 2574 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538585 2574 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538587 2574 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 29 18:34:20.543858 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538591 2574 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538593 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538598 2574 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538601 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538604 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538607 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538610 2574 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538612 2574 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538615 2574 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538618 2574 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:20.538621 2574 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.538626 2574 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.538744 2574 server.go:962] "Client rotation is on, will bootstrap in background" Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.541692 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 29 18:34:20.544341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.543071 2574 server.go:1019] "Starting client certificate rotation" Jul 29 18:34:20.544674 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.543168 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 18:34:20.544674 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.543204 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 29 18:34:20.545764 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.545750 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 18:34:20.549951 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.549913 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 29 18:34:20.563109 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.563087 2574 log.go:25] "Validated CRI v1 runtime API" Jul 29 18:34:20.565342 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.565329 2574 log.go:25] "Validated CRI v1 image API" Jul 29 18:34:20.566145 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.566129 2574 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 29 18:34:20.568968 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.568950 2574 fs.go:135] Filesystem UUIDs: map[4f64c0fa-16a9-4bd6-9c1b-1014501e65b8:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 b66a377f-6df2-4347-b14f-f7d82561da0d:/dev/nvme0n1p4] Jul 29 18:34:20.569034 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.568968 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 29 18:34:20.575854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.575726 2574 manager.go:216] Machine: {Timestamp:2026-07-29 18:34:20.574511634 +0000 UTC m=+0.287423115 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099503 MemoryCapacity:32812171264 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27893370602f9620bff512926325ca SystemUUID:ec278933-7060-2f96-20bf-f512926325ca BootID:66349120-79c2-4425-9f5e-b2b5266410bb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:5a:b8:15:5f:5d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:5a:b8:15:5f:5d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8a:5d:51:ee:ae:3a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812171264 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 29 18:34:20.575854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.575843 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 29 18:34:20.575985 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.575961 2574 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 29 18:34:20.576940 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.576891 2574 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 29 18:34:20.577083 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.576944 2574 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-210.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 29 18:34:20.577655 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.577644 2574 topology_manager.go:138] "Creating topology manager with none policy" Jul 29 18:34:20.577693 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.577658 2574 container_manager_linux.go:309] "Creating device plugin manager" Jul 29 18:34:20.577693 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.577666 2574 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 18:34:20.578227 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.578214 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 29 18:34:20.579756 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.579739 2574 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 29 18:34:20.580859 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.580841 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 18:34:20.583177 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583164 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 29 18:34:20.583302 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583291 2574 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 29 18:34:20.583343 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583323 2574 kubelet.go:486] "Attempting to sync node with API server" Jul 29 18:34:20.583343 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583334 2574 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 29 18:34:20.583394 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583344 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 29 18:34:20.583394 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583357 2574 kubelet.go:398] "Adding apiserver pod source" Jul 29 18:34:20.583394 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583366 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 29 18:34:20.583761 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583738 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 18:34:20.583807 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.583770 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 29 18:34:20.589190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.589169 2574 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 29 18:34:20.590611 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.590597 2574 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 29 18:34:20.590663 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.590628 2574 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 29 18:34:20.591348 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591336 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 29 18:34:20.591386 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591357 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 29 18:34:20.591386 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591366 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 29 18:34:20.591386 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591373 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 29 18:34:20.591386 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591382 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 29 18:34:20.591386 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591387 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 29 18:34:20.591509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591396 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 29 18:34:20.591509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591402 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 29 18:34:20.591509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591409 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 29 18:34:20.591509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591414 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 29 18:34:20.591509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591428 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 29 18:34:20.591509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591433 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 29 18:34:20.591901 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591892 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 29 18:34:20.591901 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.591902 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 29 18:34:20.595355 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.595342 2574 server.go:1268] "Started kubelet" Jul 29 18:34:20.595477 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.595411 2574 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 29 18:34:20.595530 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.595438 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 29 18:34:20.595591 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.595578 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 29 18:34:20.596444 ip-10-0-137-210 systemd[1]: Started Kubernetes Kubelet. Jul 29 18:34:20.597403 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.597295 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-210.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 29 18:34:20.597403 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.597373 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 29 18:34:20.597403 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.597355 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 29 18:34:20.597551 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.597426 2574 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 29 18:34:20.598342 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.598328 2574 server.go:310] "Adding debug handlers to kubelet server" Jul 29 18:34:20.600251 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.600232 2574 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 29 18:34:20.603053 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.602419 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-210.ec2.internal.18c6d66bc1817b5a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-210.ec2.internal,UID:ip-10-0-137-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-210.ec2.internal,},FirstTimestamp:2026-07-29 18:34:20.595313498 +0000 UTC m=+0.308224981,LastTimestamp:2026-07-29 18:34:20.595313498 +0000 UTC m=+0.308224981,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-210.ec2.internal,}" Jul 29 18:34:20.605622 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.605603 2574 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 29 18:34:20.605622 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.605614 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 29 18:34:20.605773 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.605717 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:20.606114 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.606093 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 29 18:34:20.606227 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.606197 2574 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 29 18:34:20.606286 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.606229 2574 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 29 18:34:20.606442 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.606423 2574 reconstruct.go:97] "Volume reconstruction finished" Jul 29 18:34:20.606442 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.606441 2574 reconciler.go:29] "Reconciler: start to sync state" Jul 29 18:34:20.607494 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.607436 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 29 18:34:20.607494 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.607474 2574 factory.go:55] Registering systemd factory Jul 29 18:34:20.607494 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.607484 2574 factory.go:223] Registration of the systemd container factory successfully Jul 29 18:34:20.608084 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.608067 2574 factory.go:153] Registering CRI-O factory Jul 29 18:34:20.608084 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.608085 2574 factory.go:223] Registration of the crio container factory successfully Jul 29 18:34:20.608226 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.608108 2574 factory.go:103] Registering Raw factory Jul 29 18:34:20.608226 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.608128 2574 manager.go:1196] Started watching for new ooms in manager Jul 29 18:34:20.608616 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.608602 2574 manager.go:349] Starting recovery of all containers Jul 29 18:34:20.611710 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.611665 2574 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-210.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 29 18:34:20.615159 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.615124 2574 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 29 18:34:20.616497 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.616319 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2zckg" Jul 29 18:34:20.620153 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.620133 2574 manager.go:354] Recovery completed Jul 29 18:34:20.624561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.624548 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.626422 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.626406 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.626485 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.626436 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.626485 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.626446 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.627027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627013 2574 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 29 18:34:20.627027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627021 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.627027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627025 2574 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 29 18:34:20.627124 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627040 2574 state_mem.go:36] "Initialized new in-memory state store" Jul 29 18:34:20.627932 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627892 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.628068 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627969 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.628068 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.627988 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.628310 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.628302 2574 policy_none.go:49] "None policy: Start" Jul 29 18:34:20.628362 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.628316 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 29 18:34:20.628362 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.628328 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 29 18:34:20.628937 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.628831 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-210.ec2.internal.18c6d66bc35c293a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-210.ec2.internal,UID:ip-10-0-137-210.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-210.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-210.ec2.internal,},FirstTimestamp:2026-07-29 18:34:20.626422074 +0000 UTC m=+0.339333558,LastTimestamp:2026-07-29 18:34:20.626422074 +0000 UTC m=+0.339333558,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-210.ec2.internal,}" Jul 29 18:34:20.629492 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.629480 2574 policy_none.go:47] "Start" Jul 29 18:34:20.630066 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.630049 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-2zckg" Jul 29 18:34:20.672558 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.672531 2574 manager.go:337] "Starting Device Plugin manager" Jul 29 18:34:20.672697 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.672579 2574 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 29 18:34:20.672697 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.672595 2574 server.go:87] "Starting device plugin registration server" Jul 29 18:34:20.672880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.672868 2574 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 29 18:34:20.672945 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.672884 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 29 18:34:20.673011 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.672993 2574 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 29 18:34:20.673096 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.673079 2574 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 29 18:34:20.673096 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.673093 2574 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 29 18:34:20.673633 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.673609 2574 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 29 18:34:20.673709 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.673661 2574 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:20.759234 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.759183 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 29 18:34:20.760614 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.760594 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 29 18:34:20.760664 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.760641 2574 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 29 18:34:20.760701 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.760669 2574 kubelet.go:2448] "Starting kubelet main sync loop" Jul 29 18:34:20.760738 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.760710 2574 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 29 18:34:20.763430 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.763410 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:20.773620 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.773562 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.774466 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.774450 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.774543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.774479 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.774543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.774490 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.774543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.774513 2574 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.782982 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.782960 2574 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.783075 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.782987 2574 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-210.ec2.internal\": node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:20.800421 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.800393 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:20.861412 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.861365 2574 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal"] Jul 29 18:34:20.861574 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.861466 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.863086 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.863071 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.863178 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.863103 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.863178 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.863115 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.864456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.864443 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.864640 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.864603 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.864695 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.864641 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.865167 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.865144 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.865167 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.865160 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.865167 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.865174 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.865369 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.865182 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.865369 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.865194 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.865369 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.865183 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.866237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.866217 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.866320 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.866247 2574 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 29 18:34:20.866871 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.866856 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientMemory" Jul 29 18:34:20.866959 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.866886 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasNoDiskPressure" Jul 29 18:34:20.866959 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.866899 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeHasSufficientPID" Jul 29 18:34:20.889674 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.889648 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-210.ec2.internal\" not found" node="ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.894283 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.894264 2574 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-210.ec2.internal\" not found" node="ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.901411 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:20.901381 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:20.907688 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.907665 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37538435c645a6e5590e800e1cbd349a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-210.ec2.internal\" (UID: \"37538435c645a6e5590e800e1cbd349a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.907801 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.907700 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7f67fffa56adfd6cd3b6df937507fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal\" (UID: \"e7f67fffa56adfd6cd3b6df937507fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:20.907801 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:20.907735 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7f67fffa56adfd6cd3b6df937507fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal\" (UID: \"e7f67fffa56adfd6cd3b6df937507fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.002302 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.002269 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.008308 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.008284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7f67fffa56adfd6cd3b6df937507fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal\" (UID: \"e7f67fffa56adfd6cd3b6df937507fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.008382 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.008314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7f67fffa56adfd6cd3b6df937507fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal\" (UID: \"e7f67fffa56adfd6cd3b6df937507fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.008382 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.008332 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37538435c645a6e5590e800e1cbd349a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-210.ec2.internal\" (UID: \"37538435c645a6e5590e800e1cbd349a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.008451 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.008384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/37538435c645a6e5590e800e1cbd349a-config\") pod \"kube-apiserver-proxy-ip-10-0-137-210.ec2.internal\" (UID: \"37538435c645a6e5590e800e1cbd349a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.008451 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.008399 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e7f67fffa56adfd6cd3b6df937507fef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal\" (UID: \"e7f67fffa56adfd6cd3b6df937507fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.008451 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.008384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7f67fffa56adfd6cd3b6df937507fef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal\" (UID: \"e7f67fffa56adfd6cd3b6df937507fef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.102959 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.102856 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.192486 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.192446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.198628 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.198568 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.203447 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.203428 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.304021 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.303978 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.404551 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.404467 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.505036 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.505003 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.542518 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.542487 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 29 18:34:21.543159 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.542655 2574 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 29 18:34:21.605361 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.605331 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.634006 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.633964 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-28 18:29:20 +0000 UTC" deadline="2028-01-13 19:52:36.633156027 +0000 UTC" Jul 29 18:34:21.634006 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.633995 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12793h18m14.999163846s" Jul 29 18:34:21.663521 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:21.663489 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37538435c645a6e5590e800e1cbd349a.slice/crio-1c69c3f99e066911b64aa0925f06e9bc6e085b34725fb52ebfc9bab2a89109a7 WatchSource:0}: Error finding container 1c69c3f99e066911b64aa0925f06e9bc6e085b34725fb52ebfc9bab2a89109a7: Status 404 returned error can't find the container with id 1c69c3f99e066911b64aa0925f06e9bc6e085b34725fb52ebfc9bab2a89109a7 Jul 29 18:34:21.664755 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:21.664721 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7f67fffa56adfd6cd3b6df937507fef.slice/crio-2e4380e4e522043f2fb1b5ce3855d98256276d78beb365a71f9728990fdc0338 WatchSource:0}: Error finding container 2e4380e4e522043f2fb1b5ce3855d98256276d78beb365a71f9728990fdc0338: Status 404 returned error can't find the container with id 2e4380e4e522043f2fb1b5ce3855d98256276d78beb365a71f9728990fdc0338 Jul 29 18:34:21.674442 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.674420 2574 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:21.675037 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.675019 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:34:21.706842 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:21.706794 2574 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-210.ec2.internal\" not found" Jul 29 18:34:21.755139 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.755111 2574 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:21.764336 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.764279 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" event={"ID":"37538435c645a6e5590e800e1cbd349a","Type":"ContainerStarted","Data":"1c69c3f99e066911b64aa0925f06e9bc6e085b34725fb52ebfc9bab2a89109a7"} Jul 29 18:34:21.765463 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.765436 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" event={"ID":"e7f67fffa56adfd6cd3b6df937507fef","Type":"ContainerStarted","Data":"2e4380e4e522043f2fb1b5ce3855d98256276d78beb365a71f9728990fdc0338"} Jul 29 18:34:21.799598 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.799569 2574 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:21.806412 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.806388 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.817562 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.817538 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 18:34:21.818533 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.818519 2574 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" Jul 29 18:34:21.829565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:21.829543 2574 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 29 18:34:22.412068 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.412023 2574 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 29 18:34:22.585023 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.584986 2574 apiserver.go:52] "Watching apiserver" Jul 29 18:34:22.593619 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.593588 2574 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 29 18:34:22.595044 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.594987 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p","openshift-image-registry/node-ca-lb7g7","openshift-multus/multus-additional-cni-plugins-wzs89","openshift-multus/network-metrics-daemon-blt95","openshift-network-diagnostics/network-check-target-mm74h","openshift-ovn-kubernetes/ovnkube-node-bncm6","kube-system/global-pull-secret-syncer-lzz4p","kube-system/konnectivity-agent-qhf4c","openshift-cluster-node-tuning-operator/tuned-hn87z","openshift-dns/node-resolver-zp82l","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal","openshift-multus/multus-c6z64","openshift-network-operator/iptables-alerter-whl6j","kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal"] Jul 29 18:34:22.596871 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.596844 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.598190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.598165 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.600318 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.599436 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.600318 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.599848 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-wgzfr\"" Jul 29 18:34:22.600318 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.600119 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.600746 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.600805 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.601057 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.601087 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.601159 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.601089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ckvns\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.601813 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 29 18:34:22.602036 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.602041 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.602543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.602521 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.602664 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.602606 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:22.603535 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.603511 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tvhvm\"" Jul 29 18:34:22.604103 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.604084 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.604282 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.604268 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 29 18:34:22.604665 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.604638 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 29 18:34:22.605342 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.605198 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.607305 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.606639 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.607305 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.607089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cvhs5\"" Jul 29 18:34:22.609049 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.607909 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 29 18:34:22.609049 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.608531 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 29 18:34:22.609049 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.608585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.609717 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.609692 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-zhk5w\"" Jul 29 18:34:22.609817 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.609785 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 29 18:34:22.612117 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.611828 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.612276 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.612254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:22.612370 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.612338 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:22.612587 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.612564 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 29 18:34:22.612709 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.612692 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 29 18:34:22.612890 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.612873 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 29 18:34:22.612965 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.612890 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 29 18:34:22.613155 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613139 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-czv7p\"" Jul 29 18:34:22.613235 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613174 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.613380 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613364 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.613513 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613498 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 29 18:34:22.613580 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613530 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 29 18:34:22.613580 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613564 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.613682 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613657 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.613757 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613743 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8blk5\"" Jul 29 18:34:22.613820 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.613791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:22.613870 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.613831 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:22.614919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.614756 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbddp\"" Jul 29 18:34:22.614919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.614856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.615700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.615298 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.615700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.615610 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-hostroot\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-socket-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616306 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-os-release\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616334 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysctl-conf\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616386 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-systemd\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.616456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/880d8492-9c09-471e-a21e-c05da619503c-cni-binary-copy\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616460 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-ovnkube-config\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/595efa78-ec09-4a14-a58f-49396e741db4-ovn-node-metrics-cert\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616514 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-sys-fs\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616569 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0-agent-certs\") pod \"konnectivity-agent-qhf4c\" (UID: \"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0\") " pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616596 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-kubernetes\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616620 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-var-lib-kubelet\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616652 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-socket-dir-parent\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-cni-multus\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-kubelet\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616731 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-cni-bin\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616754 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-run-netns\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-ovn\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616808 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq7tm\" (UniqueName: \"kubernetes.io/projected/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-kube-api-access-lq7tm\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616861 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aee5f3b0-2217-4539-b351-afec36c4fb22-kubelet-config\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-multus-certs\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616916 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-systemd\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616933 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0-konnectivity-ca\") pod \"konnectivity-agent-qhf4c\" (UID: \"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0\") " pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysconfig\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.616979 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-run\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617021 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-cnibin\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617043 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-etc-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617079 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617153 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aee5f3b0-2217-4539-b351-afec36c4fb22-dbus\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617189 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-cnibin\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-conf-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.617590 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617261 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-etc-kubernetes\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-ovnkube-script-lib\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617293 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-k8s-cni-cncf-io\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617294 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-systemd-units\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617340 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-registration-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617407 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-os-release\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617422 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-cni-netd\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617438 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-lib-modules\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-kubelet\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617483 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-log-socket\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617498 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b68d62f-17b5-4815-85f7-3f528e7cb62f-tmp\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617517 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6tbs\" (UniqueName: \"kubernetes.io/projected/52e090a8-414b-41e4-a580-b5cd3bb2adaa-kube-api-access-w6tbs\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-system-cni-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysctl-d\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-sys\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.618408 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-tuned\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617614 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52e090a8-414b-41e4-a580-b5cd3bb2adaa-iptables-alerter-script\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617623 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-l4fsz\"" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-cni-bin\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617673 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-var-lib-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617691 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617707 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-env-overrides\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617728 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-host\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617744 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhpct\" (UniqueName: \"kubernetes.io/projected/fa2699c4-0096-4475-b291-17a166b857cd-kube-api-access-lhpct\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617764 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-netns\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-slash\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617813 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-node-log\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617830 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mwp2\" (UniqueName: \"kubernetes.io/projected/595efa78-ec09-4a14-a58f-49396e741db4-kube-api-access-9mwp2\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617837 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617854 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-device-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-modprobe-d\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.619190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxj8g\" (UniqueName: \"kubernetes.io/projected/8b68d62f-17b5-4815-85f7-3f528e7cb62f-kube-api-access-hxj8g\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52e090a8-414b-41e4-a580-b5cd3bb2adaa-host-slash\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617927 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/880d8492-9c09-471e-a21e-c05da619503c-multus-daemon-config\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617952 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xjv8\" (UniqueName: \"kubernetes.io/projected/880d8492-9c09-471e-a21e-c05da619503c-kube-api-access-5xjv8\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.617976 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-run-ovn-kubernetes\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.618020 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-etc-selinux\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.618086 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 29 18:34:22.619958 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.618076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-cni-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.626192 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.626172 2574 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 29 18:34:22.648074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.648048 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7flm" Jul 29 18:34:22.653893 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.653864 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-l7flm" Jul 29 18:34:22.707879 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.707848 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 29 18:34:22.718339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718293 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mwp2\" (UniqueName: \"kubernetes.io/projected/595efa78-ec09-4a14-a58f-49396e741db4-kube-api-access-9mwp2\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.718339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-device-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.718560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-modprobe-d\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.718560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-device-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.718560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718512 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-modprobe-d\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.718560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718515 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxj8g\" (UniqueName: \"kubernetes.io/projected/8b68d62f-17b5-4815-85f7-3f528e7cb62f-kube-api-access-hxj8g\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.718560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718547 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52e090a8-414b-41e4-a580-b5cd3bb2adaa-host-slash\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718573 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/880d8492-9c09-471e-a21e-c05da619503c-multus-daemon-config\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718601 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xjv8\" (UniqueName: \"kubernetes.io/projected/880d8492-9c09-471e-a21e-c05da619503c-kube-api-access-5xjv8\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718628 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-run-ovn-kubernetes\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/52e090a8-414b-41e4-a580-b5cd3bb2adaa-host-slash\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718694 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-etc-selinux\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb2c2975-b7bc-489d-85cd-831a7d745acc-hosts-file\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-cni-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-hostroot\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-socket-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.718824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718830 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-os-release\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718856 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718885 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkks\" (UniqueName: \"kubernetes.io/projected/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-kube-api-access-6nkks\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718915 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysctl-conf\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.718921 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-run-ovn-kubernetes\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719009 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-hostroot\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719048 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-systemd\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719079 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/880d8492-9c09-471e-a21e-c05da619503c-cni-binary-copy\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-os-release\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-ovnkube-config\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/595efa78-ec09-4a14-a58f-49396e741db4-ovn-node-metrics-cert\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-etc-selinux\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719158 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-systemd\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719097 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysctl-conf\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719218 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-sys-fs\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-sys-fs\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0-agent-certs\") pod \"konnectivity-agent-qhf4c\" (UID: \"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0\") " pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.719366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2c2975-b7bc-489d-85cd-831a7d745acc-tmp-dir\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/880d8492-9c09-471e-a21e-c05da619503c-multus-daemon-config\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719341 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-kubernetes\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-var-lib-kubelet\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-cni-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719417 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-var-lib-kubelet\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719313 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-socket-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719533 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-kubernetes\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719585 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-socket-dir-parent\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-cni-multus\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-cni-binary-copy\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719650 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-socket-dir-parent\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719662 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-cni-multus\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719679 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-kubelet\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719654 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-kubelet\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719699 2574 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-cni-bin\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719704 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-cni-bin\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.720270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719748 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs2tl\" (UniqueName: \"kubernetes.io/projected/d9930a6b-bf85-4439-aa7e-f78c277fb37f-kube-api-access-rs2tl\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719765 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-run-netns\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-ovn\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/880d8492-9c09-471e-a21e-c05da619503c-cni-binary-copy\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719797 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq7tm\" (UniqueName: \"kubernetes.io/projected/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-kube-api-access-lq7tm\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-ovnkube-config\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719824 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719853 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719854 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-run-netns\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-ovn\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aee5f3b0-2217-4539-b351-afec36c4fb22-kubelet-config\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719940 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-multus-certs\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719966 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-systemd\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719989 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0-konnectivity-ca\") pod \"konnectivity-agent-qhf4c\" (UID: \"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0\") " pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.719982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aee5f3b0-2217-4539-b351-afec36c4fb22-kubelet-config\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720017 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-host\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-systemd\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.721175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysconfig\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-multus-certs\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720131 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-run\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-cnibin\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysconfig\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720179 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-run\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720206 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720219 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-cnibin\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-etc-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720321 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-etc-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720366 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aee5f3b0-2217-4539-b351-afec36c4fb22-dbus\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-tuning-conf-dir\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-cnibin\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720476 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-conf-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720486 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-cnibin\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720495 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0-konnectivity-ca\") pod \"konnectivity-agent-qhf4c\" (UID: \"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0\") " pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720505 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-etc-kubernetes\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720526 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aee5f3b0-2217-4539-b351-afec36c4fb22-dbus\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720544 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-multus-conf-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.720550 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720538 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-ovnkube-script-lib\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720577 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-etc-kubernetes\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-k8s-cni-cncf-io\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.720686 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:23.220663593 +0000 UTC m=+2.933575081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720696 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-k8s-cni-cncf-io\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-systemd-units\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720734 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-registration-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720759 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-os-release\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-cni-netd\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.722919 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720843 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-lib-modules\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720866 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-kubelet\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720902 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-log-socket\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720925 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fa2699c4-0096-4475-b291-17a166b857cd-system-cni-dir\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720927 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b68d62f-17b5-4815-85f7-3f528e7cb62f-tmp\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720967 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6tbs\" (UniqueName: \"kubernetes.io/projected/52e090a8-414b-41e4-a580-b5cd3bb2adaa-kube-api-access-w6tbs\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.720962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-ovnkube-script-lib\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-system-cni-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm8fq\" (UniqueName: \"kubernetes.io/projected/fb2c2975-b7bc-489d-85cd-831a7d745acc-kube-api-access-bm8fq\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721074 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-lib-modules\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-log-socket\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721113 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-cni-netd\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-os-release\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721123 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-serviceca\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721146 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-systemd-units\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.723653 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721163 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-system-cni-dir\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysctl-d\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-sys\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721237 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-sysctl-d\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721246 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-sys\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-tuned\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52e090a8-414b-41e4-a580-b5cd3bb2adaa-iptables-alerter-script\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721318 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-cni-bin\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721356 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-cni-bin\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-var-lib-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721398 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-env-overrides\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721450 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-host\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhpct\" (UniqueName: \"kubernetes.io/projected/fa2699c4-0096-4475-b291-17a166b857cd-kube-api-access-lhpct\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-netns\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-slash\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721290 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-registration-dir\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.724329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-var-lib-kubelet\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-node-log\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721640 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8b68d62f-17b5-4815-85f7-3f528e7cb62f-host\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721654 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-node-log\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721691 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/880d8492-9c09-471e-a21e-c05da619503c-host-run-netns\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-slash\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721747 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-run-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/52e090a8-414b-41e4-a580-b5cd3bb2adaa-iptables-alerter-script\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721400 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-var-lib-openvswitch\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721791 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/595efa78-ec09-4a14-a58f-49396e741db4-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.721838 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/595efa78-ec09-4a14-a58f-49396e741db4-env-overrides\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.722394 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/fa2699c4-0096-4475-b291-17a166b857cd-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.723803 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/8b68d62f-17b5-4815-85f7-3f528e7cb62f-etc-tuned\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.723808 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/595efa78-ec09-4a14-a58f-49396e741db4-ovn-node-metrics-cert\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.725054 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.723936 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0-agent-certs\") pod \"konnectivity-agent-qhf4c\" (UID: \"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0\") " pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.728229 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.728190 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxj8g\" (UniqueName: \"kubernetes.io/projected/8b68d62f-17b5-4815-85f7-3f528e7cb62f-kube-api-access-hxj8g\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.729031 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.728982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xjv8\" (UniqueName: \"kubernetes.io/projected/880d8492-9c09-471e-a21e-c05da619503c-kube-api-access-5xjv8\") pod \"multus-c6z64\" (UID: \"880d8492-9c09-471e-a21e-c05da619503c\") " pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.729787 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.729764 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq7tm\" (UniqueName: \"kubernetes.io/projected/99afb60f-0a6a-4c96-be0f-c0e3a2193de5-kube-api-access-lq7tm\") pod \"aws-ebs-csi-driver-node-db74p\" (UID: \"99afb60f-0a6a-4c96-be0f-c0e3a2193de5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.730210 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.730191 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mwp2\" (UniqueName: \"kubernetes.io/projected/595efa78-ec09-4a14-a58f-49396e741db4-kube-api-access-9mwp2\") pod \"ovnkube-node-bncm6\" (UID: \"595efa78-ec09-4a14-a58f-49396e741db4\") " pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.731933 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.731899 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b68d62f-17b5-4815-85f7-3f528e7cb62f-tmp\") pod \"tuned-hn87z\" (UID: \"8b68d62f-17b5-4815-85f7-3f528e7cb62f\") " pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.738482 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.738458 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhpct\" (UniqueName: \"kubernetes.io/projected/fa2699c4-0096-4475-b291-17a166b857cd-kube-api-access-lhpct\") pod \"multus-additional-cni-plugins-wzs89\" (UID: \"fa2699c4-0096-4475-b291-17a166b857cd\") " pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.738654 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.738628 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6tbs\" (UniqueName: \"kubernetes.io/projected/52e090a8-414b-41e4-a580-b5cd3bb2adaa-kube-api-access-w6tbs\") pod \"iptables-alerter-whl6j\" (UID: \"52e090a8-414b-41e4-a580-b5cd3bb2adaa\") " pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.822193 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm8fq\" (UniqueName: \"kubernetes.io/projected/fb2c2975-b7bc-489d-85cd-831a7d745acc-kube-api-access-bm8fq\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822226 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-serviceca\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb2c2975-b7bc-489d-85cd-831a7d745acc-hosts-file\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822286 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkks\" (UniqueName: \"kubernetes.io/projected/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-kube-api-access-6nkks\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822311 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2c2975-b7bc-489d-85cd-831a7d745acc-tmp-dir\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822340 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs2tl\" (UniqueName: \"kubernetes.io/projected/d9930a6b-bf85-4439-aa7e-f78c277fb37f-kube-api-access-rs2tl\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:22.822374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:22.822809 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822384 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/fb2c2975-b7bc-489d-85cd-831a7d745acc-hosts-file\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.822809 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822446 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-host\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.822809 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-host\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.822809 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.822568 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:22.822809 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.822651 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:23.322627846 +0000 UTC m=+3.035539334 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:22.822809 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822707 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fb2c2975-b7bc-489d-85cd-831a7d745acc-tmp-dir\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.823133 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.822834 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-serviceca\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.829537 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.829500 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:22.829537 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.829526 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:22.829537 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.829543 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:22.829809 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:22.829614 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:23.329593781 +0000 UTC m=+3.042505269 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:22.832804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.832772 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkks\" (UniqueName: \"kubernetes.io/projected/d8b39bd2-2b35-4acd-bb0d-8f70ef7af169-kube-api-access-6nkks\") pod \"node-ca-lb7g7\" (UID: \"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169\") " pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:22.833290 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.833267 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs2tl\" (UniqueName: \"kubernetes.io/projected/d9930a6b-bf85-4439-aa7e-f78c277fb37f-kube-api-access-rs2tl\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:22.834581 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.834553 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm8fq\" (UniqueName: \"kubernetes.io/projected/fb2c2975-b7bc-489d-85cd-831a7d745acc-kube-api-access-bm8fq\") pod \"node-resolver-zp82l\" (UID: \"fb2c2975-b7bc-489d-85cd-831a7d745acc\") " pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.912634 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.912589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-hn87z" Jul 29 18:34:22.924461 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.924382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-whl6j" Jul 29 18:34:22.934021 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.933992 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-wzs89" Jul 29 18:34:22.941892 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.941862 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-c6z64" Jul 29 18:34:22.949082 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.949049 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:22.957209 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.957174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" Jul 29 18:34:22.966374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.966346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:22.974098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.974070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-zp82l" Jul 29 18:34:22.980277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:22.980251 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lb7g7" Jul 29 18:34:23.227297 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.227192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:23.227470 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.227386 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:23.227530 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.227479 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:24.227455068 +0000 UTC m=+3.940366553 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:23.328344 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.328305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:23.328525 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.328462 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:23.328608 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.328539 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:24.328518505 +0000 UTC m=+4.041429990 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:23.429737 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.429690 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:23.429967 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.429899 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:23.429967 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.429950 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:23.429967 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.429965 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:23.430132 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.430040 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:24.430018949 +0000 UTC m=+4.142930437 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:23.646302 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:23.646258 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52e090a8_414b_41e4_a580_b5cd3bb2adaa.slice/crio-14d4f75e4717f72d7961db6a284082409232e21e93bc7ec57043156bfe609775 WatchSource:0}: Error finding container 14d4f75e4717f72d7961db6a284082409232e21e93bc7ec57043156bfe609775: Status 404 returned error can't find the container with id 14d4f75e4717f72d7961db6a284082409232e21e93bc7ec57043156bfe609775 Jul 29 18:34:23.648535 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:23.648068 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ad76cc7_9aa2_4664_bcb1_c00ffbbc3af0.slice/crio-73448fd791f173eba683ef17dd01e86333480d4ec806309fd2034bf6a42acee3 WatchSource:0}: Error finding container 73448fd791f173eba683ef17dd01e86333480d4ec806309fd2034bf6a42acee3: Status 404 returned error can't find the container with id 73448fd791f173eba683ef17dd01e86333480d4ec806309fd2034bf6a42acee3 Jul 29 18:34:23.648961 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:23.648922 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595efa78_ec09_4a14_a58f_49396e741db4.slice/crio-c3532985306f8605be826f72cf571d518aad610651ec3259888e7353de3ef674 WatchSource:0}: Error finding container c3532985306f8605be826f72cf571d518aad610651ec3259888e7353de3ef674: Status 404 returned error can't find the container with id c3532985306f8605be826f72cf571d518aad610651ec3259888e7353de3ef674 Jul 29 18:34:23.649860 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:34:23.649825 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod880d8492_9c09_471e_a21e_c05da619503c.slice/crio-3a4c67a2b3bc3597ebec875b9faf8f57429680db25d6c847b30389d95c11a72e WatchSource:0}: Error finding container 3a4c67a2b3bc3597ebec875b9faf8f57429680db25d6c847b30389d95c11a72e: Status 404 returned error can't find the container with id 3a4c67a2b3bc3597ebec875b9faf8f57429680db25d6c847b30389d95c11a72e Jul 29 18:34:23.656394 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.656350 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:22 +0000 UTC" deadline="2028-05-15 02:33:03.893084516 +0000 UTC" Jul 29 18:34:23.656468 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.656395 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15727h58m40.236693263s" Jul 29 18:34:23.761416 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.761208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:23.761575 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.761261 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:23.761575 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.761528 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:23.761651 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:23.761588 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:23.771245 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.771218 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhf4c" event={"ID":"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0","Type":"ContainerStarted","Data":"73448fd791f173eba683ef17dd01e86333480d4ec806309fd2034bf6a42acee3"} Jul 29 18:34:23.774058 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.774034 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whl6j" event={"ID":"52e090a8-414b-41e4-a580-b5cd3bb2adaa","Type":"ContainerStarted","Data":"14d4f75e4717f72d7961db6a284082409232e21e93bc7ec57043156bfe609775"} Jul 29 18:34:23.775960 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.775921 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" event={"ID":"37538435c645a6e5590e800e1cbd349a","Type":"ContainerStarted","Data":"29cde50a75dae1e3af47cad762d53fad7d239d903445b0b49224a5349f00739f"} Jul 29 18:34:23.777403 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.777384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerStarted","Data":"ced7cfd43496307ae205d144876627691a8cf875c3f425bbc762af96b5db35b9"} Jul 29 18:34:23.779154 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.779125 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hn87z" event={"ID":"8b68d62f-17b5-4815-85f7-3f528e7cb62f","Type":"ContainerStarted","Data":"71f038eeb7092a3beb7ad6c270b8e5a93ad72e6539c584cbd0c4a9ba1b30c678"} Jul 29 18:34:23.780395 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.780371 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"c3532985306f8605be826f72cf571d518aad610651ec3259888e7353de3ef674"} Jul 29 18:34:23.781522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.781504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c6z64" event={"ID":"880d8492-9c09-471e-a21e-c05da619503c","Type":"ContainerStarted","Data":"3a4c67a2b3bc3597ebec875b9faf8f57429680db25d6c847b30389d95c11a72e"} Jul 29 18:34:23.786635 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.786614 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" event={"ID":"99afb60f-0a6a-4c96-be0f-c0e3a2193de5","Type":"ContainerStarted","Data":"eb4c48be80d19e05493357b3c464028befc2411bc6e722fe07290bd62845dc10"} Jul 29 18:34:23.787569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.787549 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zp82l" event={"ID":"fb2c2975-b7bc-489d-85cd-831a7d745acc","Type":"ContainerStarted","Data":"f4130d406f31a9620b0dc2d97e98c94ccab709aec96c52093c7982ca56bb1baf"} Jul 29 18:34:23.789112 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.789095 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lb7g7" event={"ID":"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169","Type":"ContainerStarted","Data":"2626349b3c4d2c6d8b2aa7ab14fa36f275af3835b77100345721b0e9760868f0"} Jul 29 18:34:23.790397 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:23.790350 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-210.ec2.internal" podStartSLOduration=2.790338435 podStartE2EDuration="2.790338435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:23.790231956 +0000 UTC m=+3.503143444" watchObservedRunningTime="2026-07-29 18:34:23.790338435 +0000 UTC m=+3.503249926" Jul 29 18:34:24.236323 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.236231 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:24.236497 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.236422 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:24.236497 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.236494 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:26.236471997 +0000 UTC m=+5.949383470 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:24.337321 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.337282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:24.337495 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.337484 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:24.337555 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.337550 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:26.337529563 +0000 UTC m=+6.050441035 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:24.438677 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.438564 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:24.438855 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.438781 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:24.438855 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.438810 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:24.438855 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.438824 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:24.439135 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.438886 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:26.438866413 +0000 UTC m=+6.151777898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:24.656931 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.656811 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-28 18:29:22 +0000 UTC" deadline="2028-01-31 14:59:08.690308716 +0000 UTC" Jul 29 18:34:24.656931 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.656880 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13220h24m44.033432426s" Jul 29 18:34:24.761956 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.761643 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:24.761956 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:24.761773 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:24.793833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.793792 2574 generic.go:356] "Generic (PLEG): container finished" podID="e7f67fffa56adfd6cd3b6df937507fef" containerID="39dba593991f5dc6513012248274afa1be51da659f295c9c80b90d1ee1210660" exitCode=0 Jul 29 18:34:24.794846 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:24.794819 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" event={"ID":"e7f67fffa56adfd6cd3b6df937507fef","Type":"ContainerDied","Data":"39dba593991f5dc6513012248274afa1be51da659f295c9c80b90d1ee1210660"} Jul 29 18:34:25.762020 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:25.761987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:25.762484 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:25.762119 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:25.762484 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:25.762210 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:25.762484 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:25.762286 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:25.801677 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:25.801643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" event={"ID":"e7f67fffa56adfd6cd3b6df937507fef","Type":"ContainerStarted","Data":"7d42eb6f18983f4f685e7805ba982e287a72ef5739129600bc897d6ae7b0c4da"} Jul 29 18:34:25.818088 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:25.818020 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-210.ec2.internal" podStartSLOduration=4.817997554 podStartE2EDuration="4.817997554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:25.817221024 +0000 UTC m=+5.530132515" watchObservedRunningTime="2026-07-29 18:34:25.817997554 +0000 UTC m=+5.530909046" Jul 29 18:34:26.253943 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:26.253847 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:26.254109 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.254023 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:26.254109 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.254085 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:30.254066426 +0000 UTC m=+9.966977908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:26.354455 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:26.354419 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:26.354637 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.354554 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:26.354637 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.354597 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:30.354584128 +0000 UTC m=+10.067495595 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:26.455136 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:26.455100 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:26.455296 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.455264 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:26.455296 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.455284 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:26.455422 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.455306 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:26.455422 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.455363 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:30.455345343 +0000 UTC m=+10.168256817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:26.763814 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:26.763781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:26.764317 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:26.763953 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:27.762588 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:27.761846 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:27.762588 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:27.761996 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:27.762588 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:27.762442 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:27.762588 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:27.762538 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:28.763110 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:28.763058 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:28.763662 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:28.763202 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:29.761531 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:29.761034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:29.761531 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:29.761076 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:29.761531 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:29.761181 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:29.761531 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:29.761322 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:30.291105 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:30.291061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:30.291564 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.291222 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:30.291564 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.291287 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.291266424 +0000 UTC m=+18.004177896 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:30.392100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:30.392062 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:30.392267 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.392248 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:30.392345 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.392330 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.392305437 +0000 UTC m=+18.105216923 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:30.493388 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:30.493348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:30.493543 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.493519 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:30.493613 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.493549 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:30.493613 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.493563 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:30.493721 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.493634 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:38.493612362 +0000 UTC m=+18.206523847 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:30.762391 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:30.762263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:30.762391 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:30.762368 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:31.761873 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:31.761823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:31.762277 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:31.761984 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:31.762397 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:31.762373 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:31.762528 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:31.762476 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:32.762130 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:32.761964 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:32.762583 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:32.762548 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:33.761070 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:33.761035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:33.761266 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:33.761163 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:33.761266 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:33.761221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:33.761373 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:33.761344 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:34.761854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:34.761823 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:34.762280 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:34.761990 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:35.760972 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:35.760944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:35.761166 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:35.760950 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:35.761166 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:35.761069 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:35.761166 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:35.761134 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:36.761871 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:36.761841 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:36.762326 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:36.762010 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:37.761119 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:37.761084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:37.761318 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:37.761084 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:37.761318 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:37.761220 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:37.761318 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:37.761284 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:38.351287 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:38.351249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:38.351806 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.351397 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:38.351806 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.351469 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:34:54.351451002 +0000 UTC m=+34.064362474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:38.452448 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:38.452407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:38.452627 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.452563 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:38.452689 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.452630 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:34:54.452615646 +0000 UTC m=+34.165527113 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:38.553155 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:38.553112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:38.553410 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.553283 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:38.553410 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.553300 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:38.553410 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.553312 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:38.553410 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.553382 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:34:54.553367895 +0000 UTC m=+34.266279364 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:38.764492 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:38.764466 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:38.764676 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:38.764583 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:39.761228 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:39.761190 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:39.761228 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:39.761214 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:39.761732 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:39.761331 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:39.761732 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:39.761409 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:40.762882 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:40.762839 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:40.763294 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:40.762997 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:41.761822 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:41.761788 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:41.762010 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:41.761789 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:41.762010 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:41.761903 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:41.762010 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:41.761980 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:42.762983 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.761669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:42.762983 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:42.761955 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:42.843647 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.842870 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-hn87z" event={"ID":"8b68d62f-17b5-4815-85f7-3f528e7cb62f","Type":"ContainerStarted","Data":"7319354a2daa9814b572cb00133221efcf0e9ed5daeb954dae68cd151dc9f289"} Jul 29 18:34:42.848281 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.848245 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-c6z64" event={"ID":"880d8492-9c09-471e-a21e-c05da619503c","Type":"ContainerStarted","Data":"d2f7f07ee791feb34197197019c86ddab3204ab73c33b60b8bc9ab59e9680e46"} Jul 29 18:34:42.850469 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.850440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" event={"ID":"99afb60f-0a6a-4c96-be0f-c0e3a2193de5","Type":"ContainerStarted","Data":"997b39753f6d24d60d0775552ca22388007a6cd9f9270db661db51f18404d6c9"} Jul 29 18:34:42.858940 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.858881 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lb7g7" event={"ID":"d8b39bd2-2b35-4acd-bb0d-8f70ef7af169","Type":"ContainerStarted","Data":"4d4216b78121fe51f5148262389e9b5514811e6eb0a0656bc40ab356ec3819e3"} Jul 29 18:34:42.861399 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.861365 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-qhf4c" event={"ID":"2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0","Type":"ContainerStarted","Data":"18ac22bb250f3722f6a5e2e2c65b4652a04e84ee0deb7421da28076f81b2bba2"} Jul 29 18:34:42.861762 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.861727 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-hn87z" podStartSLOduration=3.869555745 podStartE2EDuration="22.861716031s" totalImagesPullingTime="18.992160286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:42.861471725 +0000 UTC m=+22.574383228" watchObservedRunningTime="2026-07-29 18:34:42.861716031 +0000 UTC m=+22.574627513" Jul 29 18:34:42.863048 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.863031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerStarted","Data":"bea7b1e8dfa51f7148c38fe4f815933c9b7e3357ddce72c383fe1c0d83a364ae"} Jul 29 18:34:42.879647 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.879568 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-c6z64" podStartSLOduration=3.842032207 podStartE2EDuration="22.879550922s" totalImagesPullingTime="19.037518715s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:42.879474684 +0000 UTC m=+22.592386174" watchObservedRunningTime="2026-07-29 18:34:42.879550922 +0000 UTC m=+22.592462412" Jul 29 18:34:42.926072 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.925866 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-qhf4c" podStartSLOduration=2.945407822 podStartE2EDuration="21.925850153s" totalImagesPullingTime="18.980442331s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:42.925649683 +0000 UTC m=+22.638561175" watchObservedRunningTime="2026-07-29 18:34:42.925850153 +0000 UTC m=+22.638761642" Jul 29 18:34:42.951869 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.951823 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-zp82l" podStartSLOduration=2.979169871 podStartE2EDuration="21.95180636s" totalImagesPullingTime="18.972636489s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:42.951355905 +0000 UTC m=+22.664267398" watchObservedRunningTime="2026-07-29 18:34:42.95180636 +0000 UTC m=+22.664717850" Jul 29 18:34:42.966527 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:42.966483 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lb7g7" podStartSLOduration=2.986355864 podStartE2EDuration="21.966467244s" totalImagesPullingTime="18.98011138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:42.966256197 +0000 UTC m=+22.679167688" watchObservedRunningTime="2026-07-29 18:34:42.966467244 +0000 UTC m=+22.679378735" Jul 29 18:34:43.418411 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.418385 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:43.418939 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.418895 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:43.761771 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.761737 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:43.761965 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.761738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:43.761965 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:43.761852 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:43.762056 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:43.761962 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:43.867438 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.867400 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-zp82l" event={"ID":"fb2c2975-b7bc-489d-85cd-831a7d745acc","Type":"ContainerStarted","Data":"ad10abefcca6ed2e19b1a6194054e22e6b6411183997bc9e03ee6d30b9971ce2"} Jul 29 18:34:43.868798 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.868767 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2699c4-0096-4475-b291-17a166b857cd" containerID="bea7b1e8dfa51f7148c38fe4f815933c9b7e3357ddce72c383fe1c0d83a364ae" exitCode=0 Jul 29 18:34:43.868887 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.868852 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerDied","Data":"bea7b1e8dfa51f7148c38fe4f815933c9b7e3357ddce72c383fe1c0d83a364ae"} Jul 29 18:34:43.872546 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.871957 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"03ab66cdeb3c99d7b9e4a1713b29e40959c6f1411e4e7f636b443b281649ed7e"} Jul 29 18:34:43.872546 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.872027 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"11c2061cda9444a55eee3e460b72d95cca1f30adfd4f201b28280dc89c52053e"} Jul 29 18:34:43.872546 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.872061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"248884ee9853fde9f3e2deefbc9fd3b8052abe3e79fd9266f16df36aa9e07729"} Jul 29 18:34:43.872546 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.872096 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"5e7dc0b8a4852228e5233671a5958f7c2add561e0a813c6809ae9410f764a80f"} Jul 29 18:34:43.872546 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.872127 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"6234b16c26866245240ae2931eccda4f48bce0839de1edb1fc2802740439f49b"} Jul 29 18:34:43.872546 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:43.872176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"21e70522c6e88ce0ceb3d2773e44488bb4350aa8e7fbdbc0b0bab1d876b12187"} Jul 29 18:34:44.134065 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.134029 2574 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 29 18:34:44.690645 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.690527 2574 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-29T18:34:44.134057308Z","UUID":"d8c7c79c-2cbf-40aa-b72d-a1a26de187a9","Handler":null,"Name":"","Endpoint":""} Jul 29 18:34:44.692269 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.692246 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 29 18:34:44.692393 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.692292 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 29 18:34:44.765257 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.765229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:44.765426 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:44.765378 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:44.876338 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.876296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" event={"ID":"99afb60f-0a6a-4c96-be0f-c0e3a2193de5","Type":"ContainerStarted","Data":"558638d040bbbafed80dfa040afe54d298ace859b57c70c26abd2e691b521ed5"} Jul 29 18:34:44.879196 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.879172 2574 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 29 18:34:44.879196 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.879182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-whl6j" event={"ID":"52e090a8-414b-41e4-a580-b5cd3bb2adaa","Type":"ContainerStarted","Data":"4f626252cce49fce816526dffd409468352d3b03efd9f0f348bf30c913e9fed8"} Jul 29 18:34:44.894753 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:44.894686 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-whl6j" podStartSLOduration=5.910272347 podStartE2EDuration="24.89467132s" totalImagesPullingTime="18.984398973s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:44.894586791 +0000 UTC m=+24.607498280" watchObservedRunningTime="2026-07-29 18:34:44.89467132 +0000 UTC m=+24.607582811" Jul 29 18:34:45.761668 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:45.761586 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:45.761668 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:45.761626 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:45.761887 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:45.761714 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:45.761887 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:45.761841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:45.879405 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:45.879368 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:45.880008 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:45.879987 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-qhf4c" Jul 29 18:34:45.887969 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:45.887936 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"a333be34d7d47ce65720af1eca3c26ce2b39476886b69ab454696f745c7f70a5"} Jul 29 18:34:46.761208 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:46.761010 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:46.761387 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:46.761308 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:46.892736 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:46.892699 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" event={"ID":"99afb60f-0a6a-4c96-be0f-c0e3a2193de5","Type":"ContainerStarted","Data":"0767398d0e9f943d8ad4236988a585260221698cc72ad42d166540b2603d53bf"} Jul 29 18:34:47.761093 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:47.761061 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:47.761093 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:47.761094 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:47.761370 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:47.761183 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:47.761370 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:47.761315 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:48.762112 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.761903 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:48.762761 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:48.762209 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:48.899289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.899250 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2699c4-0096-4475-b291-17a166b857cd" containerID="13865ec03005ee38cbe237d11618d9515fb1c1d597255d8fe8e1378e6ead0321" exitCode=0 Jul 29 18:34:48.899488 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.899313 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerDied","Data":"13865ec03005ee38cbe237d11618d9515fb1c1d597255d8fe8e1378e6ead0321"} Jul 29 18:34:48.903196 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.903174 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" event={"ID":"595efa78-ec09-4a14-a58f-49396e741db4","Type":"ContainerStarted","Data":"72cddc84622160657a4615ace7115435e9bd567a9bf35d061d91aa02e0f9ee56"} Jul 29 18:34:48.903435 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.903409 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:48.919747 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.919726 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:48.931986 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.931894 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-db74p" podStartSLOduration=5.59216425 podStartE2EDuration="27.931873097s" totalImagesPullingTime="22.339708847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:46.911149588 +0000 UTC m=+26.624061078" watchObservedRunningTime="2026-07-29 18:34:48.931873097 +0000 UTC m=+28.644784587" Jul 29 18:34:48.955878 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:48.955822 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" podStartSLOduration=8.905662386 podStartE2EDuration="27.955806263s" totalImagesPullingTime="19.050143877s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:48.954802514 +0000 UTC m=+28.667714004" watchObservedRunningTime="2026-07-29 18:34:48.955806263 +0000 UTC m=+28.668717752" Jul 29 18:34:49.222710 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.222638 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zp82l_fb2c2975-b7bc-489d-85cd-831a7d745acc/dns-node-resolver/0.log" Jul 29 18:34:49.761123 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.761077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:49.761440 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.761077 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:49.761440 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:49.761232 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:49.761440 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:49.761313 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:49.797249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.797225 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lb7g7_d8b39bd2-2b35-4acd-bb0d-8f70ef7af169/node-ca/0.log" Jul 29 18:34:49.906687 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.906661 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:49.906888 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.906873 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:49.927824 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:49.927796 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:34:50.039588 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.039560 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lzz4p"] Jul 29 18:34:50.039738 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.039679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:50.039818 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:50.039799 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:50.042850 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.042814 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mm74h"] Jul 29 18:34:50.043004 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.042972 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:50.043076 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:50.043057 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:50.043581 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.043556 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-blt95"] Jul 29 18:34:50.043689 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.043668 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:50.043799 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:50.043777 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:50.909964 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.909905 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2699c4-0096-4475-b291-17a166b857cd" containerID="9faebf52ffae7c6545f4cb50eaab6371536e547492ab5961955bcbf81ee1eb6a" exitCode=0 Jul 29 18:34:50.909964 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:50.909954 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerDied","Data":"9faebf52ffae7c6545f4cb50eaab6371536e547492ab5961955bcbf81ee1eb6a"} Jul 29 18:34:51.761768 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:51.761739 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:51.761939 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:51.761782 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:51.761939 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:51.761833 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:51.761939 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:51.761850 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:51.762101 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:51.761934 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:51.762101 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:51.761981 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:53.761969 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:53.761781 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:53.762400 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:53.761841 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:53.762400 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:53.762042 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:53.762400 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:53.762116 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:53.762400 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:53.761840 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:53.762400 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:53.762197 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:53.920080 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:53.920052 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2699c4-0096-4475-b291-17a166b857cd" containerID="bd7f9e7006ba47062dbe980fa76062c9746d8aeed8dc4b602a30c95e68e6d716" exitCode=0 Jul 29 18:34:53.920242 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:53.920102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerDied","Data":"bd7f9e7006ba47062dbe980fa76062c9746d8aeed8dc4b602a30c95e68e6d716"} Jul 29 18:34:54.372978 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:54.372892 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:54.373137 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.373025 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:54.373137 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.373081 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret podName:aee5f3b0-2217-4539-b351-afec36c4fb22 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:26.373066899 +0000 UTC m=+66.085978369 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret") pod "global-pull-secret-syncer-lzz4p" (UID: "aee5f3b0-2217-4539-b351-afec36c4fb22") : object "kube-system"/"original-pull-secret" not registered Jul 29 18:34:54.473747 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:54.473716 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:54.473909 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.473835 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:54.473991 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.473908 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs podName:d9930a6b-bf85-4439-aa7e-f78c277fb37f nodeName:}" failed. No retries permitted until 2026-07-29 18:35:26.473889282 +0000 UTC m=+66.186800759 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs") pod "network-metrics-daemon-blt95" (UID: "d9930a6b-bf85-4439-aa7e-f78c277fb37f") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 29 18:34:54.574827 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:54.574788 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:54.575032 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.574944 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 29 18:34:54.575032 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.574970 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 29 18:34:54.575032 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.574982 2574 projected.go:196] Error preparing data for projected volume kube-api-access-556fq for pod openshift-network-diagnostics/network-check-target-mm74h: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:54.575188 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:54.575041 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq podName:f5e5ca28-cb02-43bd-8ca5-907cd845ba1c nodeName:}" failed. No retries permitted until 2026-07-29 18:35:26.575022938 +0000 UTC m=+66.287934420 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-556fq" (UniqueName: "kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq") pod "network-check-target-mm74h" (UID: "f5e5ca28-cb02-43bd-8ca5-907cd845ba1c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 29 18:34:55.761097 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:55.761069 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:55.761533 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:55.761179 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:55.761533 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:55.761190 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:55.761533 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:55.761362 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:55.761533 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:55.761431 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:55.761533 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:55.761352 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:57.761356 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:57.761322 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:57.761796 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:57.761324 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:57.761796 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:57.761447 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:57.761796 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:57.761336 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:57.761796 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:57.761503 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:57.761796 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:57.761605 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:57.931013 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:57.930985 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerStarted","Data":"dfc04faa7a838c39789907445b077de25bdaac6540e2d3297a4d5726821eab38"} Jul 29 18:34:58.936522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:58.936491 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2699c4-0096-4475-b291-17a166b857cd" containerID="dfc04faa7a838c39789907445b077de25bdaac6540e2d3297a4d5726821eab38" exitCode=0 Jul 29 18:34:58.936522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:58.936515 2574 generic.go:356] "Generic (PLEG): container finished" podID="fa2699c4-0096-4475-b291-17a166b857cd" containerID="7ca2e5a4e8336ec2b77cc60c8fe11b352d9d500f26df3217da1b947211574308" exitCode=0 Jul 29 18:34:58.937129 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:58.936548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerDied","Data":"dfc04faa7a838c39789907445b077de25bdaac6540e2d3297a4d5726821eab38"} Jul 29 18:34:58.937129 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:58.936583 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerDied","Data":"7ca2e5a4e8336ec2b77cc60c8fe11b352d9d500f26df3217da1b947211574308"} Jul 29 18:34:59.761811 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:59.761776 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:34:59.761961 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:59.761791 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:34:59.761961 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:59.761794 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:34:59.762068 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:59.761957 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:34:59.762068 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:59.762017 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:34:59.762146 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:34:59.762102 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:34:59.942385 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:59.942354 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-wzs89" event={"ID":"fa2699c4-0096-4475-b291-17a166b857cd","Type":"ContainerStarted","Data":"7af23e0a03cfabcde4c837a37dbb4dbe50239ff9ce2823c899af55e2bb084f50"} Jul 29 18:34:59.964397 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:34:59.964348 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-wzs89" podStartSLOduration=9.653891302 podStartE2EDuration="39.964332443s" totalImagesPullingTime="30.310441141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:34:59.96276364 +0000 UTC m=+39.675675130" watchObservedRunningTime="2026-07-29 18:34:59.964332443 +0000 UTC m=+39.677243932" Jul 29 18:35:01.761294 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:01.761244 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:01.761294 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:01.761285 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:01.761772 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:01.761357 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:01.761772 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:01.761375 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:35:01.761772 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:01.761422 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:35:01.761772 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:01.761498 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:35:03.761216 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:03.761180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:03.761216 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:03.761222 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:03.761728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:03.761193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:03.761728 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:03.761302 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:35:03.761728 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:03.761393 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:35:03.761728 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:03.761468 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:35:05.761049 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:05.761017 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:05.761470 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:05.761018 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:05.761470 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:05.761117 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:35:05.761470 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:05.761027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:05.761470 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:05.761193 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:35:05.761470 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:05.761284 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:35:07.761293 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:07.761260 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:07.761836 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:07.761364 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:07.761836 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:07.761367 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:35:07.761836 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:07.761372 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:07.761836 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:07.761476 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:35:07.761836 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:07.761538 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:35:09.761697 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:09.761663 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:09.762138 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:09.761774 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:09.762138 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:09.761790 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:35:09.762138 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:09.761805 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:09.762138 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:09.761868 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:35:09.762138 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:09.761953 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:35:11.761470 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:11.761435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:11.761857 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:11.761478 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:11.761857 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:11.761529 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:11.761857 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:11.761540 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-blt95" podUID="d9930a6b-bf85-4439-aa7e-f78c277fb37f" Jul 29 18:35:11.761857 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:11.761592 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mm74h" podUID="f5e5ca28-cb02-43bd-8ca5-907cd845ba1c" Jul 29 18:35:11.761857 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:11.761647 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lzz4p" podUID="aee5f3b0-2217-4539-b351-afec36c4fb22" Jul 29 18:35:13.661328 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.661297 2574 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-210.ec2.internal" event="NodeReady" Jul 29 18:35:13.661719 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.661411 2574 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 29 18:35:13.713150 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.713123 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5f6d6d5c85-6hnmr"] Jul 29 18:35:13.732654 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.732628 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-n967v"] Jul 29 18:35:13.732827 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.732807 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.737790 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.737759 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-zc9c4\"" Jul 29 18:35:13.738213 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.738188 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 29 18:35:13.738453 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.737805 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 29 18:35:13.738613 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.738588 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 29 18:35:13.742045 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.742025 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 29 18:35:13.748616 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.748600 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6d6d5c85-6hnmr"] Jul 29 18:35:13.748693 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.748627 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-8fq5r"] Jul 29 18:35:13.748731 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.748706 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:13.751391 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.751372 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 29 18:35:13.751491 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.751476 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 29 18:35:13.751556 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.751502 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9grlj\"" Jul 29 18:35:13.751610 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.751563 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 29 18:35:13.751732 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.751719 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 29 18:35:13.766685 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.766665 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fq5r"] Jul 29 18:35:13.766767 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.766728 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n967v"] Jul 29 18:35:13.766767 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.766734 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:13.766835 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.766798 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:13.766835 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.766818 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:13.766835 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.766827 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:13.769524 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.769506 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hfcvr\"" Jul 29 18:35:13.769626 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.769538 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 18:35:13.769779 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.769766 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 18:35:13.770009 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.769993 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 18:35:13.770078 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.770054 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 18:35:13.770233 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.770213 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 29 18:35:13.770339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.770310 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfhf2\"" Jul 29 18:35:13.770406 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.770347 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2xgwr\"" Jul 29 18:35:13.770467 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.770424 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 29 18:35:13.831698 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.831669 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gn8s2"] Jul 29 18:35:13.854987 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.854964 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gn8s2"] Jul 29 18:35:13.855097 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.855062 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:13.857675 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.857656 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 29 18:35:13.857831 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.857814 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-84hqk\"" Jul 29 18:35:13.858273 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.858256 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 29 18:35:13.858528 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.858512 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 29 18:35:13.921887 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.921863 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-trusted-ca\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.921894 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8ea8d-12fc-4eba-bf4a-de072713200a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:13.922046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.921957 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-registry-certificates\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.921996 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk7ql\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-kube-api-access-pk7ql\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922038 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cf63bdd-f881-4c6e-9a12-65930772fa1d-metrics-tls\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:13.922249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaf8ea8d-12fc-4eba-bf4a-de072713200a-data-volume\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:13.922249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922095 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf63bdd-f881-4c6e-9a12-65930772fa1d-config-volume\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:13.922249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922119 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfhjg\" (UniqueName: \"kubernetes.io/projected/eaf8ea8d-12fc-4eba-bf4a-de072713200a-kube-api-access-lfhjg\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:13.922249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922216 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8ea8d-12fc-4eba-bf4a-de072713200a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:13.922249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-registry-tls\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922251 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-bound-sa-token\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922270 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf63bdd-f881-4c6e-9a12-65930772fa1d-tmp-dir\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-image-registry-private-configuration\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-installation-pull-secrets\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922353 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaf8ea8d-12fc-4eba-bf4a-de072713200a-crio-socket\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzrxv\" (UniqueName: \"kubernetes.io/projected/2cf63bdd-f881-4c6e-9a12-65930772fa1d-kube-api-access-lzrxv\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:13.922456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:13.922416 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-ca-trust-extracted\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023370 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023330 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-installation-pull-secrets\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023370 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaf8ea8d-12fc-4eba-bf4a-de072713200a-crio-socket\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.023618 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023387 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzrxv\" (UniqueName: \"kubernetes.io/projected/2cf63bdd-f881-4c6e-9a12-65930772fa1d-kube-api-access-lzrxv\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.023618 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-ca-trust-extracted\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023618 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-trusted-ca\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023618 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023513 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8ea8d-12fc-4eba-bf4a-de072713200a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.023618 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaf8ea8d-12fc-4eba-bf4a-de072713200a-crio-socket\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.023618 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023605 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-registry-certificates\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023648 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk7ql\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-kube-api-access-pk7ql\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cf63bdd-f881-4c6e-9a12-65930772fa1d-metrics-tls\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023695 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaf8ea8d-12fc-4eba-bf4a-de072713200a-data-volume\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf63bdd-f881-4c6e-9a12-65930772fa1d-config-volume\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfhjg\" (UniqueName: \"kubernetes.io/projected/eaf8ea8d-12fc-4eba-bf4a-de072713200a-kube-api-access-lfhjg\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93db8590-3846-473d-a8e1-d68e33613e1a-cert\") pod \"ingress-canary-gn8s2\" (UID: \"93db8590-3846-473d-a8e1-d68e33613e1a\") " pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8ea8d-12fc-4eba-bf4a-de072713200a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-ca-trust-extracted\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.023885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-registry-tls\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.024365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-bound-sa-token\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.024365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf63bdd-f881-4c6e-9a12-65930772fa1d-tmp-dir\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.024365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.023985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jbdh\" (UniqueName: \"kubernetes.io/projected/93db8590-3846-473d-a8e1-d68e33613e1a-kube-api-access-4jbdh\") pod \"ingress-canary-gn8s2\" (UID: \"93db8590-3846-473d-a8e1-d68e33613e1a\") " pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.024365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.024033 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-image-registry-private-configuration\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.024559 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.024516 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-registry-certificates\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.024646 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.024620 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaf8ea8d-12fc-4eba-bf4a-de072713200a-data-volume\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.024769 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.024750 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaf8ea8d-12fc-4eba-bf4a-de072713200a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.025178 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.025052 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2cf63bdd-f881-4c6e-9a12-65930772fa1d-tmp-dir\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.025178 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.025148 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2cf63bdd-f881-4c6e-9a12-65930772fa1d-config-volume\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.025982 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.025957 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-trusted-ca\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.028113 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.028066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaf8ea8d-12fc-4eba-bf4a-de072713200a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.028113 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.028103 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2cf63bdd-f881-4c6e-9a12-65930772fa1d-metrics-tls\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.028259 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.028245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-registry-tls\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.028325 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.028307 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-image-registry-private-configuration\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.028361 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.028310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-installation-pull-secrets\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.032907 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.032326 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzrxv\" (UniqueName: \"kubernetes.io/projected/2cf63bdd-f881-4c6e-9a12-65930772fa1d-kube-api-access-lzrxv\") pod \"dns-default-8fq5r\" (UID: \"2cf63bdd-f881-4c6e-9a12-65930772fa1d\") " pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.034996 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.033174 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfhjg\" (UniqueName: \"kubernetes.io/projected/eaf8ea8d-12fc-4eba-bf4a-de072713200a-kube-api-access-lfhjg\") pod \"insights-runtime-extractor-n967v\" (UID: \"eaf8ea8d-12fc-4eba-bf4a-de072713200a\") " pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.034996 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.033517 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-bound-sa-token\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.037038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.037018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk7ql\" (UniqueName: \"kubernetes.io/projected/0c5cbc72-89f0-47ca-aa3f-bb66c74d47db-kube-api-access-pk7ql\") pod \"image-registry-5f6d6d5c85-6hnmr\" (UID: \"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db\") " pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.045373 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.045346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.057599 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.057580 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-n967v" Jul 29 18:35:14.092277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.092254 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:14.124478 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.124429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jbdh\" (UniqueName: \"kubernetes.io/projected/93db8590-3846-473d-a8e1-d68e33613e1a-kube-api-access-4jbdh\") pod \"ingress-canary-gn8s2\" (UID: \"93db8590-3846-473d-a8e1-d68e33613e1a\") " pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.124620 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.124537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93db8590-3846-473d-a8e1-d68e33613e1a-cert\") pod \"ingress-canary-gn8s2\" (UID: \"93db8590-3846-473d-a8e1-d68e33613e1a\") " pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.127203 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.127184 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/93db8590-3846-473d-a8e1-d68e33613e1a-cert\") pod \"ingress-canary-gn8s2\" (UID: \"93db8590-3846-473d-a8e1-d68e33613e1a\") " pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.133955 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.133910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jbdh\" (UniqueName: \"kubernetes.io/projected/93db8590-3846-473d-a8e1-d68e33613e1a-kube-api-access-4jbdh\") pod \"ingress-canary-gn8s2\" (UID: \"93db8590-3846-473d-a8e1-d68e33613e1a\") " pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.164087 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.163654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gn8s2" Jul 29 18:35:14.268941 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.268795 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-n967v"] Jul 29 18:35:14.281136 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:14.281080 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf8ea8d_12fc_4eba_bf4a_de072713200a.slice/crio-3258ab380ed01e4e0daea604e422859cdc941a86df863fd0f3a4e07513affef8 WatchSource:0}: Error finding container 3258ab380ed01e4e0daea604e422859cdc941a86df863fd0f3a4e07513affef8: Status 404 returned error can't find the container with id 3258ab380ed01e4e0daea604e422859cdc941a86df863fd0f3a4e07513affef8 Jul 29 18:35:14.347909 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.347878 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gn8s2"] Jul 29 18:35:14.350666 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:14.350640 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93db8590_3846_473d_a8e1_d68e33613e1a.slice/crio-918c37420372e103593dc0bccffd43b7dac2b0e8b1013c184b75c09b42676e4c WatchSource:0}: Error finding container 918c37420372e103593dc0bccffd43b7dac2b0e8b1013c184b75c09b42676e4c: Status 404 returned error can't find the container with id 918c37420372e103593dc0bccffd43b7dac2b0e8b1013c184b75c09b42676e4c Jul 29 18:35:14.490007 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.489938 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-8fq5r"] Jul 29 18:35:14.492236 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.492215 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5f6d6d5c85-6hnmr"] Jul 29 18:35:14.493244 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:14.493219 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf63bdd_f881_4c6e_9a12_65930772fa1d.slice/crio-5548bb522cb7d992dd0d7c3eba64ed446c15d283e49abc9a18075721dcf1f3fc WatchSource:0}: Error finding container 5548bb522cb7d992dd0d7c3eba64ed446c15d283e49abc9a18075721dcf1f3fc: Status 404 returned error can't find the container with id 5548bb522cb7d992dd0d7c3eba64ed446c15d283e49abc9a18075721dcf1f3fc Jul 29 18:35:14.495201 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:14.495182 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5cbc72_89f0_47ca_aa3f_bb66c74d47db.slice/crio-8587ec1d390600d2fac0835c6f6ffd8cf7aa1eb92989f6fc715661b3a26beb5f WatchSource:0}: Error finding container 8587ec1d390600d2fac0835c6f6ffd8cf7aa1eb92989f6fc715661b3a26beb5f: Status 404 returned error can't find the container with id 8587ec1d390600d2fac0835c6f6ffd8cf7aa1eb92989f6fc715661b3a26beb5f Jul 29 18:35:14.978160 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.978120 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gn8s2" event={"ID":"93db8590-3846-473d-a8e1-d68e33613e1a","Type":"ContainerStarted","Data":"918c37420372e103593dc0bccffd43b7dac2b0e8b1013c184b75c09b42676e4c"} Jul 29 18:35:14.979520 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.979491 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fq5r" event={"ID":"2cf63bdd-f881-4c6e-9a12-65930772fa1d","Type":"ContainerStarted","Data":"5548bb522cb7d992dd0d7c3eba64ed446c15d283e49abc9a18075721dcf1f3fc"} Jul 29 18:35:14.981416 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.981388 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" event={"ID":"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db","Type":"ContainerStarted","Data":"8dcad47eb308a82530b607cb76bef5ed5dd6bcca4179da649d88a4ca37497c7a"} Jul 29 18:35:14.981518 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.981435 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" event={"ID":"0c5cbc72-89f0-47ca-aa3f-bb66c74d47db","Type":"ContainerStarted","Data":"8587ec1d390600d2fac0835c6f6ffd8cf7aa1eb92989f6fc715661b3a26beb5f"} Jul 29 18:35:14.981585 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.981539 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:14.982648 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:14.982628 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n967v" event={"ID":"eaf8ea8d-12fc-4eba-bf4a-de072713200a","Type":"ContainerStarted","Data":"3258ab380ed01e4e0daea604e422859cdc941a86df863fd0f3a4e07513affef8"} Jul 29 18:35:15.004418 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:15.004363 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" podStartSLOduration=3.004348048 podStartE2EDuration="3.004348048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:15.003575953 +0000 UTC m=+54.716487444" watchObservedRunningTime="2026-07-29 18:35:15.004348048 +0000 UTC m=+54.717259538" Jul 29 18:35:21.000736 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.000690 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n967v" event={"ID":"eaf8ea8d-12fc-4eba-bf4a-de072713200a","Type":"ContainerStarted","Data":"f47e12c9ff4d3b90961e7b13c83cf63cf3f7c0feadc97cc51dea3ba23533c4da"} Jul 29 18:35:21.002384 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.002350 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gn8s2" event={"ID":"93db8590-3846-473d-a8e1-d68e33613e1a","Type":"ContainerStarted","Data":"fd02af395cd2b9afec4a88627cdc372170a3efb2cd468177ac0be1379b505358"} Jul 29 18:35:21.004906 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.004882 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fq5r" event={"ID":"2cf63bdd-f881-4c6e-9a12-65930772fa1d","Type":"ContainerStarted","Data":"2ecc1e86d13ab59e9948857132599ef74807ba5df7751e8bd4ba2acbd8b648d8"} Jul 29 18:35:21.005068 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.004944 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-8fq5r" event={"ID":"2cf63bdd-f881-4c6e-9a12-65930772fa1d","Type":"ContainerStarted","Data":"da011028e3b6cec490fbafed854f5a6c36d3470ca671407f25b7b5f91967ef74"} Jul 29 18:35:21.005369 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.005350 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:21.021162 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.021118 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gn8s2" podStartSLOduration=2.355685171 podStartE2EDuration="8.021104947s" totalImagesPullingTime="5.665419776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:21.020256849 +0000 UTC m=+60.733168339" watchObservedRunningTime="2026-07-29 18:35:21.021104947 +0000 UTC m=+60.734016440" Jul 29 18:35:21.038605 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.038548 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-8fq5r" podStartSLOduration=2.515790429 podStartE2EDuration="8.038528506s" totalImagesPullingTime="5.522738077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:21.037160131 +0000 UTC m=+60.750071621" watchObservedRunningTime="2026-07-29 18:35:21.038528506 +0000 UTC m=+60.751439997" Jul 29 18:35:21.932419 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:21.932389 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bncm6" Jul 29 18:35:24.015124 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:24.015089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n967v" event={"ID":"eaf8ea8d-12fc-4eba-bf4a-de072713200a","Type":"ContainerStarted","Data":"afc0a856260d22f20e17ca387b742546c4d762d3c0e2de6d55320e8de59abedf"} Jul 29 18:35:26.425652 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.425611 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:26.428341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.428314 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 29 18:35:26.439075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.439043 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aee5f3b0-2217-4539-b351-afec36c4fb22-original-pull-secret\") pod \"global-pull-secret-syncer-lzz4p\" (UID: \"aee5f3b0-2217-4539-b351-afec36c4fb22\") " pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:26.527013 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.526975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:26.529711 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.529682 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 29 18:35:26.540576 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.540549 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d9930a6b-bf85-4439-aa7e-f78c277fb37f-metrics-certs\") pod \"network-metrics-daemon-blt95\" (UID: \"d9930a6b-bf85-4439-aa7e-f78c277fb37f\") " pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:26.628151 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.628119 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:26.630796 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.630760 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 29 18:35:26.641556 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.641529 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 29 18:35:26.652022 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.651969 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556fq\" (UniqueName: \"kubernetes.io/projected/f5e5ca28-cb02-43bd-8ca5-907cd845ba1c-kube-api-access-556fq\") pod \"network-check-target-mm74h\" (UID: \"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c\") " pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:26.679361 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.679289 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hfcvr\"" Jul 29 18:35:26.682619 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.682595 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lzz4p" Jul 29 18:35:26.687752 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.687728 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:26.690184 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.690162 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mfhf2\"" Jul 29 18:35:26.698381 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:26.698363 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-blt95" Jul 29 18:35:27.400574 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:27.400544 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-blt95"] Jul 29 18:35:27.404454 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:27.404413 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9930a6b_bf85_4439_aa7e_f78c277fb37f.slice/crio-15688bdb083adfa68db12b73f2fe5d3483f6040ed47ade0c35f61dc6cfa00882 WatchSource:0}: Error finding container 15688bdb083adfa68db12b73f2fe5d3483f6040ed47ade0c35f61dc6cfa00882: Status 404 returned error can't find the container with id 15688bdb083adfa68db12b73f2fe5d3483f6040ed47ade0c35f61dc6cfa00882 Jul 29 18:35:27.421818 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:27.421761 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lzz4p"] Jul 29 18:35:27.425967 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:27.425944 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee5f3b0_2217_4539_b351_afec36c4fb22.slice/crio-dfc01ac745e3a569e4820fbb9d7546022a567803722f158a3602b7508c0eaab3 WatchSource:0}: Error finding container dfc01ac745e3a569e4820fbb9d7546022a567803722f158a3602b7508c0eaab3: Status 404 returned error can't find the container with id dfc01ac745e3a569e4820fbb9d7546022a567803722f158a3602b7508c0eaab3 Jul 29 18:35:27.437100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:27.437078 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mm74h"] Jul 29 18:35:27.440264 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:27.440228 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5e5ca28_cb02_43bd_8ca5_907cd845ba1c.slice/crio-c919a578e983f3d0ec34531311d86aaacbc56e47b98c9b0019e6b826d30938d7 WatchSource:0}: Error finding container c919a578e983f3d0ec34531311d86aaacbc56e47b98c9b0019e6b826d30938d7: Status 404 returned error can't find the container with id c919a578e983f3d0ec34531311d86aaacbc56e47b98c9b0019e6b826d30938d7 Jul 29 18:35:28.035396 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:28.035352 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mm74h" event={"ID":"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c","Type":"ContainerStarted","Data":"c919a578e983f3d0ec34531311d86aaacbc56e47b98c9b0019e6b826d30938d7"} Jul 29 18:35:28.036772 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:28.036736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blt95" event={"ID":"d9930a6b-bf85-4439-aa7e-f78c277fb37f","Type":"ContainerStarted","Data":"15688bdb083adfa68db12b73f2fe5d3483f6040ed47ade0c35f61dc6cfa00882"} Jul 29 18:35:28.039627 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:28.039286 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-n967v" event={"ID":"eaf8ea8d-12fc-4eba-bf4a-de072713200a","Type":"ContainerStarted","Data":"3017bec3615b2d3fa1a16332382ce175a5bda1d44390cd260deb83cf142d3b9e"} Jul 29 18:35:28.040959 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:28.040933 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lzz4p" event={"ID":"aee5f3b0-2217-4539-b351-afec36c4fb22","Type":"ContainerStarted","Data":"dfc01ac745e3a569e4820fbb9d7546022a567803722f158a3602b7508c0eaab3"} Jul 29 18:35:28.059190 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:28.059139 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-n967v" podStartSLOduration=2.142181026 podStartE2EDuration="15.059120692s" totalImagesPullingTime="12.916939666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:28.058134789 +0000 UTC m=+67.771046280" watchObservedRunningTime="2026-07-29 18:35:28.059120692 +0000 UTC m=+67.772032177" Jul 29 18:35:29.161597 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.161548 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9qj4p"] Jul 29 18:35:29.178112 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.178082 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw"] Jul 29 18:35:29.178269 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.178115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.185360 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185334 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 29 18:35:29.185512 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185365 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 29 18:35:29.185512 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185459 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 29 18:35:29.185654 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185539 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn2rv\"" Jul 29 18:35:29.185654 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185598 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 29 18:35:29.185760 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185668 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 29 18:35:29.185760 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.185689 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 29 18:35:29.190403 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.189943 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.190608 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.190589 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw"] Jul 29 18:35:29.193345 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.193119 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 29 18:35:29.193679 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.193658 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 29 18:35:29.193899 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.193882 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 29 18:35:29.194347 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.194170 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-pr9c4\"" Jul 29 18:35:29.251409 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-textfile\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251554 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251414 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-wtmp\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251554 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251448 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.251554 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251494 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251554 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-sys\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251588 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-root\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251692 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-tls\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.251833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251720 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8722654c-a11e-4df1-87f2-fa6137ad83e4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.251833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251819 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.252103 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjw99\" (UniqueName: \"kubernetes.io/projected/1b347548-f005-4036-bed6-d91a0815d58d-kube-api-access-bjw99\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.252103 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.252103 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251900 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fscpj\" (UniqueName: \"kubernetes.io/projected/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-api-access-fscpj\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.252103 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251954 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b347548-f005-4036-bed6-d91a0815d58d-metrics-client-ca\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.252103 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.251984 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8722654c-a11e-4df1-87f2-fa6137ad83e4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.352957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-wtmp\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353011 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353084 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-sys\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353101 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-root\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353152 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-tls\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-wtmp\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-sys\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8722654c-a11e-4df1-87f2-fa6137ad83e4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353224 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1b347548-f005-4036-bed6-d91a0815d58d-root\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjw99\" (UniqueName: \"kubernetes.io/projected/1b347548-f005-4036-bed6-d91a0815d58d-kube-api-access-bjw99\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353326 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fscpj\" (UniqueName: \"kubernetes.io/projected/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-api-access-fscpj\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.354284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353396 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b347548-f005-4036-bed6-d91a0815d58d-metrics-client-ca\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8722654c-a11e-4df1-87f2-fa6137ad83e4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353464 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-textfile\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353755 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-textfile\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-accelerators-collector-config\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.353964 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8722654c-a11e-4df1-87f2-fa6137ad83e4-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:29.354046 2574 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:29.354103 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-tls podName:1b347548-f005-4036-bed6-d91a0815d58d nodeName:}" failed. No retries permitted until 2026-07-29 18:35:29.854085594 +0000 UTC m=+69.566997067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-tls") pod "node-exporter-9qj4p" (UID: "1b347548-f005-4036-bed6-d91a0815d58d") : secret "node-exporter-tls" not found Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.354155 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1b347548-f005-4036-bed6-d91a0815d58d-metrics-client-ca\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.354235 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.355243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.354336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/8722654c-a11e-4df1-87f2-fa6137ad83e4-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.357486 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.357440 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.359149 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.359107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.359495 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.359477 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.370479 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.370452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fscpj\" (UniqueName: \"kubernetes.io/projected/8722654c-a11e-4df1-87f2-fa6137ad83e4-kube-api-access-fscpj\") pod \"kube-state-metrics-56d9f944b4-nwttw\" (UID: \"8722654c-a11e-4df1-87f2-fa6137ad83e4\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.371811 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.371788 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjw99\" (UniqueName: \"kubernetes.io/projected/1b347548-f005-4036-bed6-d91a0815d58d-kube-api-access-bjw99\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.506381 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.506343 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" Jul 29 18:35:29.858200 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.858095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-tls\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:29.861000 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:29.860968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1b347548-f005-4036-bed6-d91a0815d58d-node-exporter-tls\") pod \"node-exporter-9qj4p\" (UID: \"1b347548-f005-4036-bed6-d91a0815d58d\") " pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:30.015989 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.015955 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:35:30.029297 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.029266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.031909 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.031874 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 29 18:35:30.032084 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032056 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 29 18:35:30.032084 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.031879 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 29 18:35:30.032289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032123 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 29 18:35:30.032289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032154 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 29 18:35:30.032289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032056 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 29 18:35:30.032441 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032311 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 29 18:35:30.032441 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032379 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 29 18:35:30.033027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032852 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-2c4tl\"" Jul 29 18:35:30.033027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.032860 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 29 18:35:30.037750 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.037713 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:35:30.090520 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.090486 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9qj4p" Jul 29 18:35:30.160791 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160697 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-out\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.160791 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.160791 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160803 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160848 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160938 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160962 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.160991 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.161035 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb98h\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-kube-api-access-nb98h\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.161082 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.161115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-volume\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.161424 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.161146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-web-config\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262097 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262141 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262196 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb98h\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-kube-api-access-nb98h\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262223 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:30.262239 2574 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-volume\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-web-config\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:30.262311 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls podName:4763c8aa-6db3-4cbe-b46a-7c4ed8480e87 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:30.762287706 +0000 UTC m=+70.475199176 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87") : secret "alertmanager-main-tls" not found Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262373 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-out\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262409 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262472 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.262561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.262507 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.263454 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:30.263380 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle podName:4763c8aa-6db3-4cbe-b46a-7c4ed8480e87 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:30.763358465 +0000 UTC m=+70.476269948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87") : configmap references non-existent config key: ca-bundle.crt Jul 29 18:35:30.263601 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.263570 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.264009 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.263860 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.265906 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.265836 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.266063 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.265955 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-web-config\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.266505 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.266196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.266505 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.266459 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-tls-assets\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.266897 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.266873 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.267082 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.267060 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-out\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.267202 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.267180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.267821 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.267795 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-volume\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.274733 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.274713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb98h\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-kube-api-access-nb98h\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.765948 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.765900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.766107 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.765973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:30.766107 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:30.766014 2574 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 29 18:35:30.766190 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:35:30.766113 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls podName:4763c8aa-6db3-4cbe-b46a-7c4ed8480e87 nodeName:}" failed. No retries permitted until 2026-07-29 18:35:31.76609718 +0000 UTC m=+71.479008648 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87") : secret "alertmanager-main-tls" not found Jul 29 18:35:30.766734 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:30.766717 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:31.010767 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:31.010733 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-8fq5r" Jul 29 18:35:31.774248 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:31.774208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:31.777136 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:31.777112 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:31.846887 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:31.846857 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:35:32.174450 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:32.174289 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b347548_f005_4036_bed6_d91a0815d58d.slice/crio-504d63fee2a11851741a649a255729588a8cc3055a67bb70ec56900a28d457d1 WatchSource:0}: Error finding container 504d63fee2a11851741a649a255729588a8cc3055a67bb70ec56900a28d457d1: Status 404 returned error can't find the container with id 504d63fee2a11851741a649a255729588a8cc3055a67bb70ec56900a28d457d1 Jul 29 18:35:32.336780 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:32.336748 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw"] Jul 29 18:35:32.342609 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:32.342566 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8722654c_a11e_4df1_87f2_fa6137ad83e4.slice/crio-167b7a0cf5db2179302eab152de35c781c98639b343541c9b23b7a3e634aa2eb WatchSource:0}: Error finding container 167b7a0cf5db2179302eab152de35c781c98639b343541c9b23b7a3e634aa2eb: Status 404 returned error can't find the container with id 167b7a0cf5db2179302eab152de35c781c98639b343541c9b23b7a3e634aa2eb Jul 29 18:35:32.370912 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:32.370549 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:35:32.373185 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:32.373160 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4763c8aa_6db3_4cbe_b46a_7c4ed8480e87.slice/crio-cf5c480cba9cbccb3ac5259c89e597b707072d735ed0cf065fdabda40f28cb42 WatchSource:0}: Error finding container cf5c480cba9cbccb3ac5259c89e597b707072d735ed0cf065fdabda40f28cb42: Status 404 returned error can't find the container with id cf5c480cba9cbccb3ac5259c89e597b707072d735ed0cf065fdabda40f28cb42 Jul 29 18:35:33.065150 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.065052 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" event={"ID":"8722654c-a11e-4df1-87f2-fa6137ad83e4","Type":"ContainerStarted","Data":"167b7a0cf5db2179302eab152de35c781c98639b343541c9b23b7a3e634aa2eb"} Jul 29 18:35:33.067099 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.067069 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mm74h" event={"ID":"f5e5ca28-cb02-43bd-8ca5-907cd845ba1c","Type":"ContainerStarted","Data":"b8a854edc460c5117c39ee6245dd9e16df10d843fd81ceb166d931fc7ace76cc"} Jul 29 18:35:33.067853 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.067798 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:35:33.070207 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.070179 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blt95" event={"ID":"d9930a6b-bf85-4439-aa7e-f78c277fb37f","Type":"ContainerStarted","Data":"848347cc6767de69dc251ddabc15cb0b88834f441c6596112c41c98ab6996354"} Jul 29 18:35:33.070324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.070233 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-blt95" event={"ID":"d9930a6b-bf85-4439-aa7e-f78c277fb37f","Type":"ContainerStarted","Data":"5dd8edf2b89d62895361151213fde767af572d97fdfc8b404987d97c88aa1a64"} Jul 29 18:35:33.072433 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.072372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lzz4p" event={"ID":"aee5f3b0-2217-4539-b351-afec36c4fb22","Type":"ContainerStarted","Data":"a2680e67f103367de5cbfc778f1f1e73cd9109f57afaf34d4a1c0e7fcd6f30f0"} Jul 29 18:35:33.074532 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.074506 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qj4p" event={"ID":"1b347548-f005-4036-bed6-d91a0815d58d","Type":"ContainerStarted","Data":"504d63fee2a11851741a649a255729588a8cc3055a67bb70ec56900a28d457d1"} Jul 29 18:35:33.075790 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.075719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"cf5c480cba9cbccb3ac5259c89e597b707072d735ed0cf065fdabda40f28cb42"} Jul 29 18:35:33.084729 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.084685 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mm74h" podStartSLOduration=67.342423499 podStartE2EDuration="1m12.084671982s" totalImagesPullingTime="4.742248483s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:33.083706763 +0000 UTC m=+72.796618253" watchObservedRunningTime="2026-07-29 18:35:33.084671982 +0000 UTC m=+72.797583472" Jul 29 18:35:33.100026 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.099960 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-blt95" podStartSLOduration=67.329254837 podStartE2EDuration="1m12.099946287s" totalImagesPullingTime="4.77069145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:33.099077554 +0000 UTC m=+72.811989046" watchObservedRunningTime="2026-07-29 18:35:33.099946287 +0000 UTC m=+72.812857776" Jul 29 18:35:33.114306 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:33.114253 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lzz4p" podStartSLOduration=67.35503091 podStartE2EDuration="1m12.114235663s" totalImagesPullingTime="4.759204753s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 18:34:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:33.114016984 +0000 UTC m=+72.826928498" watchObservedRunningTime="2026-07-29 18:35:33.114235663 +0000 UTC m=+72.827147151" Jul 29 18:35:34.082274 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.082228 2574 generic.go:356] "Generic (PLEG): container finished" podID="1b347548-f005-4036-bed6-d91a0815d58d" containerID="239527b1edea6f0acf4a96d8e590ae2655c1b0077e92a6e46f933c75c441dfef" exitCode=0 Jul 29 18:35:34.082723 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.082296 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qj4p" event={"ID":"1b347548-f005-4036-bed6-d91a0815d58d","Type":"ContainerDied","Data":"239527b1edea6f0acf4a96d8e590ae2655c1b0077e92a6e46f933c75c441dfef"} Jul 29 18:35:34.085471 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.084235 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="05fdc6bd65ff9e40455519a71a5ab765a611898afc6636f5eda11d471d2e5b1a" exitCode=0 Jul 29 18:35:34.085471 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.084326 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"05fdc6bd65ff9e40455519a71a5ab765a611898afc6636f5eda11d471d2e5b1a"} Jul 29 18:35:34.088407 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.088378 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" event={"ID":"8722654c-a11e-4df1-87f2-fa6137ad83e4","Type":"ContainerStarted","Data":"74f49e9677d1282e206478240336d4a95f95c565b175fdea881cbfad70c31eef"} Jul 29 18:35:34.088517 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.088438 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" event={"ID":"8722654c-a11e-4df1-87f2-fa6137ad83e4","Type":"ContainerStarted","Data":"76d5f04a72d8eadf7cf75f75b29dcfe760e8aac84c800bba2ebbf5844dd7ae51"} Jul 29 18:35:34.088517 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.088504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" event={"ID":"8722654c-a11e-4df1-87f2-fa6137ad83e4","Type":"ContainerStarted","Data":"2429511b3eaeac9f20820eb7107b522e9e33223dd72881e2abc45556f83af482"} Jul 29 18:35:34.153302 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.153236 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-nwttw" podStartSLOduration=3.777935636 podStartE2EDuration="5.153215355s" totalImagesPullingTime="1.375279719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:34.151782817 +0000 UTC m=+73.864694320" watchObservedRunningTime="2026-07-29 18:35:34.153215355 +0000 UTC m=+73.866126846" Jul 29 18:35:34.593802 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.593723 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-778b69bf4c-2l4bf"] Jul 29 18:35:34.596725 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.596708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.599459 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.599439 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 18:35:34.599571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.599476 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 18:35:34.599717 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.599698 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 18:35:34.599790 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.599739 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 18:35:34.599846 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.599705 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 18:35:34.599902 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.599841 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 18:35:34.600122 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.600107 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 18:35:34.600158 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.600129 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9hd9l\"" Jul 29 18:35:34.604405 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.604384 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 18:35:34.606168 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.606147 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778b69bf4c-2l4bf"] Jul 29 18:35:34.697849 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.697818 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-trusted-ca-bundle\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.698032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.697874 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-config\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.698032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.697908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-serving-cert\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.698032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.697959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-service-ca\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.698032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.697987 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-oauth-serving-cert\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.698032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.698013 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-oauth-config\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.698230 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.698062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zgmm\" (UniqueName: \"kubernetes.io/projected/a7f1d4b8-742f-444b-a4ef-dc368debab5a-kube-api-access-5zgmm\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.798682 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.798643 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-config\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.798814 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.798720 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-serving-cert\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.798814 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.798751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-service-ca\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.798814 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.798780 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-oauth-serving-cert\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.799338 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.799290 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-oauth-config\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.799460 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.799361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zgmm\" (UniqueName: \"kubernetes.io/projected/a7f1d4b8-742f-444b-a4ef-dc368debab5a-kube-api-access-5zgmm\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.799460 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.799401 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-trusted-ca-bundle\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.799582 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.799497 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-config\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.799582 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.799524 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-oauth-serving-cert\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.799856 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.799830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-service-ca\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.800175 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.800137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-trusted-ca-bundle\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.802031 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.802012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-serving-cert\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.802131 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.802066 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-oauth-config\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.809454 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.809436 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zgmm\" (UniqueName: \"kubernetes.io/projected/a7f1d4b8-742f-444b-a4ef-dc368debab5a-kube-api-access-5zgmm\") pod \"console-778b69bf4c-2l4bf\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:34.906208 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:34.906123 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:35.046099 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.046061 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-778b69bf4c-2l4bf"] Jul 29 18:35:35.049706 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:35.049680 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f1d4b8_742f_444b_a4ef_dc368debab5a.slice/crio-bfba223c9767f6c87212cacb0441c72346241c1d746c9928b88ede96a2d16f08 WatchSource:0}: Error finding container bfba223c9767f6c87212cacb0441c72346241c1d746c9928b88ede96a2d16f08: Status 404 returned error can't find the container with id bfba223c9767f6c87212cacb0441c72346241c1d746c9928b88ede96a2d16f08 Jul 29 18:35:35.093206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.093146 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b69bf4c-2l4bf" event={"ID":"a7f1d4b8-742f-444b-a4ef-dc368debab5a","Type":"ContainerStarted","Data":"bfba223c9767f6c87212cacb0441c72346241c1d746c9928b88ede96a2d16f08"} Jul 29 18:35:35.095640 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.095608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qj4p" event={"ID":"1b347548-f005-4036-bed6-d91a0815d58d","Type":"ContainerStarted","Data":"404636494b16655a496be0e1be17e2174ffcefc9d060c6bc913a22c18750bd27"} Jul 29 18:35:35.095768 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.095668 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9qj4p" event={"ID":"1b347548-f005-4036-bed6-d91a0815d58d","Type":"ContainerStarted","Data":"6f4f355f677e58424fd50a268c3fa2c4761b3339105bb153b0a64697ff7ef388"} Jul 29 18:35:35.116596 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.116539 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9qj4p" podStartSLOduration=4.485597586 podStartE2EDuration="6.116518198s" totalImagesPullingTime="1.538979346s" totalInitContainerRuntime="91.941266ms" isStatefulPod=true podCreationTimestamp="2026-07-29 18:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:35.115338729 +0000 UTC m=+74.828250219" watchObservedRunningTime="2026-07-29 18:35:35.116518198 +0000 UTC m=+74.829429690" Jul 29 18:35:35.249412 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.249380 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:35:35.253494 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.253471 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.256641 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.256471 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 18:35:35.257026 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.257006 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 18:35:35.257134 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.257117 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 18:35:35.257195 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.257164 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fj2mgd5otp9tp\"" Jul 29 18:35:35.257633 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.257612 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 18:35:35.259118 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.258183 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 18:35:35.259118 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.258250 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 18:35:35.259118 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.258267 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 18:35:35.259118 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.258599 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lz7cg\"" Jul 29 18:35:35.259118 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.258656 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 18:35:35.259118 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.258863 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 18:35:35.263553 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.260645 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 18:35:35.263553 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.260821 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 18:35:35.263553 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.261075 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 18:35:35.263553 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.263385 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 18:35:35.263905 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.263879 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 18:35:35.267022 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.266998 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 18:35:35.270788 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.270766 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:35:35.303685 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303647 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.303854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.303854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303726 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.303854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.303854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.303854 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.303955 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304008 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304062 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304164 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304220 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304266 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304330 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-config\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304380 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.304700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.304399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6t4g\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-kube-api-access-m6t4g\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.404823 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.404786 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405029 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.404831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405029 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.404992 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405153 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405032 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405153 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405049 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405153 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405304 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405212 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405304 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405255 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405304 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405304 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405614 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405586 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405714 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405638 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405714 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405673 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405714 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405700 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.405950 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.405970 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-config\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6t4g\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-kube-api-access-m6t4g\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406186 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406072 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406186 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406186 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406344 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406180 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406344 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406225 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.406846 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.406818 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.408183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.408145 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.408364 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.408340 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.408728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.408699 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.409642 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.408968 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.409642 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.409073 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.409642 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.409098 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.409642 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.409601 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.410025 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.409972 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-web-config\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.410025 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.410000 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.410233 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.410089 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.410757 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.410695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.411487 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.411464 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.411881 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.411857 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-config\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.412316 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.412297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-config-out\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.412499 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.412481 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.416853 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.416835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6t4g\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-kube-api-access-m6t4g\") pod \"prometheus-k8s-0\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.567822 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.567732 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:35:35.742504 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.742465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:35:35.747613 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:35:35.747582 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff667192_20d6_4975_96ca_777b28d5c25a.slice/crio-7da88407f7de26274e020acefe9912009bda9ea5c3c991a71677e082081d3a21 WatchSource:0}: Error finding container 7da88407f7de26274e020acefe9912009bda9ea5c3c991a71677e082081d3a21: Status 404 returned error can't find the container with id 7da88407f7de26274e020acefe9912009bda9ea5c3c991a71677e082081d3a21 Jul 29 18:35:35.994651 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:35.993822 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5f6d6d5c85-6hnmr" Jul 29 18:35:36.110728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.110462 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"50e3b09d0f90fc6b103a6904ca8cf9662a2174e84740e3f49674290a601774a9"} Jul 29 18:35:36.110728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.110555 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"639db965af9d8d0c0ef4c5bedb127d83088d104d20c8d592608de646d36a42d1"} Jul 29 18:35:36.110728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.110603 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"69bec21e7052be6127ae1094886bf617beecc2028cba816d3d3d32ec02ab434a"} Jul 29 18:35:36.110728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.110647 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"c59508c07f253bdfda7396fc5ad0c900b03024e44e96649e4daef6c430e33235"} Jul 29 18:35:36.115950 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.114548 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" exitCode=0 Jul 29 18:35:36.116530 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.116211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} Jul 29 18:35:36.116530 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:36.116305 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"7da88407f7de26274e020acefe9912009bda9ea5c3c991a71677e082081d3a21"} Jul 29 18:35:37.123948 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:37.123814 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"8a6714ad69a0dd51d56ba17b0147f78002bf57de9128c300364d41c85052f6ac"} Jul 29 18:35:40.140680 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:40.140643 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerStarted","Data":"ab1f847dd5cd1370e885cf2161ea25b39ce9d00804933a71ba108d447ad03674"} Jul 29 18:35:40.142969 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:40.142945 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} Jul 29 18:35:40.143099 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:40.143001 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} Jul 29 18:35:40.144179 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:40.144162 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b69bf4c-2l4bf" event={"ID":"a7f1d4b8-742f-444b-a4ef-dc368debab5a","Type":"ContainerStarted","Data":"ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480"} Jul 29 18:35:40.175106 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:40.175048 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.790589797 podStartE2EDuration="11.175026304s" totalImagesPullingTime="6.361152461s" totalInitContainerRuntime="23.284046ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.173216453 +0000 UTC m=+79.886127943" watchObservedRunningTime="2026-07-29 18:35:40.175026304 +0000 UTC m=+79.887937812" Jul 29 18:35:40.192889 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:40.192829 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-778b69bf4c-2l4bf" podStartSLOduration=1.596097501 podStartE2EDuration="6.192805577s" totalImagesPullingTime="4.596708076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:40.191172984 +0000 UTC m=+79.904084473" watchObservedRunningTime="2026-07-29 18:35:40.192805577 +0000 UTC m=+79.905717070" Jul 29 18:35:42.154666 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:42.154631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} Jul 29 18:35:42.155078 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:42.154696 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} Jul 29 18:35:42.155078 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:42.154723 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} Jul 29 18:35:42.155078 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:42.154749 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerStarted","Data":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} Jul 29 18:35:42.185086 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:42.185009 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.9844340649999999 podStartE2EDuration="7.184992869s" totalImagesPullingTime="5.173980153s" totalInitContainerRuntime="26.578651ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:35:35 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:35:42.183000278 +0000 UTC m=+81.895911770" watchObservedRunningTime="2026-07-29 18:35:42.184992869 +0000 UTC m=+81.897904382" Jul 29 18:35:44.906319 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:44.906283 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:44.906692 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:44.906341 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:44.910989 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:44.910970 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:45.169050 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:45.169027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:35:45.568384 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:35:45.568284 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:05.098696 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:05.098668 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mm74h" Jul 29 18:36:35.568319 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:35.568279 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:35.587479 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:35.587456 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:36.373006 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:36.372973 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:49.214291 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214260 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:36:49.214816 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214567 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="alertmanager" containerID="cri-o://c59508c07f253bdfda7396fc5ad0c900b03024e44e96649e4daef6c430e33235" gracePeriod=120 Jul 29 18:36:49.214816 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214629 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-metric" containerID="cri-o://8a6714ad69a0dd51d56ba17b0147f78002bf57de9128c300364d41c85052f6ac" gracePeriod=120 Jul 29 18:36:49.214816 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214649 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-web" containerID="cri-o://639db965af9d8d0c0ef4c5bedb127d83088d104d20c8d592608de646d36a42d1" gracePeriod=120 Jul 29 18:36:49.214816 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214718 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="prom-label-proxy" containerID="cri-o://ab1f847dd5cd1370e885cf2161ea25b39ce9d00804933a71ba108d447ad03674" gracePeriod=120 Jul 29 18:36:49.214816 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214785 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy" containerID="cri-o://50e3b09d0f90fc6b103a6904ca8cf9662a2174e84740e3f49674290a601774a9" gracePeriod=120 Jul 29 18:36:49.215111 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.214718 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="config-reloader" containerID="cri-o://69bec21e7052be6127ae1094886bf617beecc2028cba816d3d3d32ec02ab434a" gracePeriod=120 Jul 29 18:36:49.408413 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408381 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="ab1f847dd5cd1370e885cf2161ea25b39ce9d00804933a71ba108d447ad03674" exitCode=0 Jul 29 18:36:49.408413 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408407 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="8a6714ad69a0dd51d56ba17b0147f78002bf57de9128c300364d41c85052f6ac" exitCode=0 Jul 29 18:36:49.408413 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408415 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="50e3b09d0f90fc6b103a6904ca8cf9662a2174e84740e3f49674290a601774a9" exitCode=0 Jul 29 18:36:49.408413 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408422 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="69bec21e7052be6127ae1094886bf617beecc2028cba816d3d3d32ec02ab434a" exitCode=0 Jul 29 18:36:49.408673 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408429 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="c59508c07f253bdfda7396fc5ad0c900b03024e44e96649e4daef6c430e33235" exitCode=0 Jul 29 18:36:49.408673 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408451 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"ab1f847dd5cd1370e885cf2161ea25b39ce9d00804933a71ba108d447ad03674"} Jul 29 18:36:49.408673 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408515 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"8a6714ad69a0dd51d56ba17b0147f78002bf57de9128c300364d41c85052f6ac"} Jul 29 18:36:49.408673 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408541 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"50e3b09d0f90fc6b103a6904ca8cf9662a2174e84740e3f49674290a601774a9"} Jul 29 18:36:49.408673 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408565 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"69bec21e7052be6127ae1094886bf617beecc2028cba816d3d3d32ec02ab434a"} Jul 29 18:36:49.408673 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:49.408589 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"c59508c07f253bdfda7396fc5ad0c900b03024e44e96649e4daef6c430e33235"} Jul 29 18:36:50.416369 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.416325 2574 generic.go:356] "Generic (PLEG): container finished" podID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerID="639db965af9d8d0c0ef4c5bedb127d83088d104d20c8d592608de646d36a42d1" exitCode=0 Jul 29 18:36:50.416722 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.416394 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"639db965af9d8d0c0ef4c5bedb127d83088d104d20c8d592608de646d36a42d1"} Jul 29 18:36:50.461500 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.461478 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:50.558715 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558608 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-out\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.558715 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558655 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-metrics-client-ca\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.558715 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558679 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-tls-assets\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.558715 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558715 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558845 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-main-db\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558887 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558913 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-volume\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558964 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.558990 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-web-config\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559074 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.559017 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nb98h\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-kube-api-access-nb98h\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559361 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.559094 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-web\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559361 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.559125 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-cluster-tls-config\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.559361 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.559163 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-metric\") pod \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\" (UID: \"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87\") " Jul 29 18:36:50.562461 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.559118 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:50.562607 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.559203 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:36:50.562607 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.562277 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.562607 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.562366 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:50.562607 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.562407 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.562607 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.562522 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.562871 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.562623 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-out" (OuterVolumeSpecName: "config-out") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:36:50.562871 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.562855 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:50.563176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.563149 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.564282 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.564259 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-kube-api-access-nb98h" (OuterVolumeSpecName: "kube-api-access-nb98h") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "kube-api-access-nb98h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:50.564421 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.564405 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-volume" (OuterVolumeSpecName: "config-volume") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.566324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.566298 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.573446 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.573424 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-web-config" (OuterVolumeSpecName: "web-config") pod "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" (UID: "4763c8aa-6db3-4cbe-b46a-7c4ed8480e87"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:50.660480 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660450 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660480 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660476 2574 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-cluster-tls-config\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660480 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660486 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660495 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-out\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660503 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-metrics-client-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660512 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-tls-assets\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660520 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-main-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660529 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-main-db\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660537 2574 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660545 2574 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-config-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660553 2574 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660560 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-web-config\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:50.660707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:50.660569 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nb98h\" (UniqueName: \"kubernetes.io/projected/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87-kube-api-access-nb98h\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:51.423534 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.423502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"4763c8aa-6db3-4cbe-b46a-7c4ed8480e87","Type":"ContainerDied","Data":"cf5c480cba9cbccb3ac5259c89e597b707072d735ed0cf065fdabda40f28cb42"} Jul 29 18:36:51.423938 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.423549 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 29 18:36:51.423938 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.423578 2574 scope.go:117] "RemoveContainer" containerID="ab1f847dd5cd1370e885cf2161ea25b39ce9d00804933a71ba108d447ad03674" Jul 29 18:36:51.431152 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.431126 2574 scope.go:117] "RemoveContainer" containerID="8a6714ad69a0dd51d56ba17b0147f78002bf57de9128c300364d41c85052f6ac" Jul 29 18:36:51.437786 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.437771 2574 scope.go:117] "RemoveContainer" containerID="50e3b09d0f90fc6b103a6904ca8cf9662a2174e84740e3f49674290a601774a9" Jul 29 18:36:51.444252 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.444233 2574 scope.go:117] "RemoveContainer" containerID="639db965af9d8d0c0ef4c5bedb127d83088d104d20c8d592608de646d36a42d1" Jul 29 18:36:51.448769 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.448747 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:36:51.451541 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.451524 2574 scope.go:117] "RemoveContainer" containerID="69bec21e7052be6127ae1094886bf617beecc2028cba816d3d3d32ec02ab434a" Jul 29 18:36:51.458397 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.458373 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 29 18:36:51.459727 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.459711 2574 scope.go:117] "RemoveContainer" containerID="c59508c07f253bdfda7396fc5ad0c900b03024e44e96649e4daef6c430e33235" Jul 29 18:36:51.466821 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:51.466802 2574 scope.go:117] "RemoveContainer" containerID="05fdc6bd65ff9e40455519a71a5ab765a611898afc6636f5eda11d471d2e5b1a" Jul 29 18:36:52.766116 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:52.766086 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" path="/var/lib/kubelet/pods/4763c8aa-6db3-4cbe-b46a-7c4ed8480e87/volumes" Jul 29 18:36:53.599483 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599447 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:53.599772 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599733 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="prometheus" containerID="cri-o://47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" gracePeriod=600 Jul 29 18:36:53.599899 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599776 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy" containerID="cri-o://a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" gracePeriod=600 Jul 29 18:36:53.599899 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599789 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-thanos" containerID="cri-o://91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" gracePeriod=600 Jul 29 18:36:53.599899 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599804 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="thanos-sidecar" containerID="cri-o://75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" gracePeriod=600 Jul 29 18:36:53.599899 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599817 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-web" containerID="cri-o://e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" gracePeriod=600 Jul 29 18:36:53.600117 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.599828 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="config-reloader" containerID="cri-o://0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" gracePeriod=600 Jul 29 18:36:53.834341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.834319 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:53.986346 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986307 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-tls-assets\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986368 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986408 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-web-config\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986430 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986462 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-trusted-ca-bundle\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986487 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-kube-rbac-proxy\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986807 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986780 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-tls\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986876 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986836 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m6t4g\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-kube-api-access-m6t4g\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986973 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986875 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-thanos-prometheus-http-client-file\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.986973 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986901 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.986973 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-0\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.986979 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-1\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-config\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987033 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-config-out\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987065 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-serving-certs-ca-bundle\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987102 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-db\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987527 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987168 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-kubelet-serving-ca-bundle\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987527 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987331 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-metrics-client-ca\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987527 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987374 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-metrics-client-certs\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987527 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987402 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-grpc-tls\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987527 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987427 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-2\") pod \"ff667192-20d6-4975-96ca-777b28d5c25a\" (UID: \"ff667192-20d6-4975-96ca-777b28d5c25a\") " Jul 29 18:36:53.987780 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.987663 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:53.988137 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.988109 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.989380 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989353 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:53.989487 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989443 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.989549 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989493 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.989634 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989609 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-kube-api-access-m6t4g" (OuterVolumeSpecName: "kube-api-access-m6t4g") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "kube-api-access-m6t4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:36:53.990125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989741 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.990125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989821 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.990125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.989854 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.990125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.990055 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.990367 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.990230 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.990666 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.990601 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.990725 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.990600 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.990985 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.990948 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:36:53.991690 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.991645 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:36:53.992000 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.991975 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.992453 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.992431 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.992545 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.992529 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-config" (OuterVolumeSpecName: "config") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:53.992753 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.992731 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-config-out" (OuterVolumeSpecName: "config-out") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:36:53.999087 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:53.999066 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-web-config" (OuterVolumeSpecName: "web-config") pod "ff667192-20d6-4975-96ca-777b28d5c25a" (UID: "ff667192-20d6-4975-96ca-777b28d5c25a"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:36:54.088765 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088737 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088784 2574 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-web-config\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088795 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088806 2574 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-kube-rbac-proxy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088815 2574 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088823 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6t4g\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-kube-api-access-m6t4g\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088832 2574 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088840 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088849 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088857 2574 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-config\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088864 2574 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-config-out\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088874 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088882 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-db\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088890 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.088895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088898 2574 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-configmap-metrics-client-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.089324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088906 2574 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-metrics-client-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.089324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088936 2574 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/ff667192-20d6-4975-96ca-777b28d5c25a-secret-grpc-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.089324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088945 2574 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/ff667192-20d6-4975-96ca-777b28d5c25a-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.089324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.088953 2574 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff667192-20d6-4975-96ca-777b28d5c25a-tls-assets\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:36:54.437663 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437633 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" exitCode=0 Jul 29 18:36:54.437663 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437659 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" exitCode=0 Jul 29 18:36:54.437663 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437665 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" exitCode=0 Jul 29 18:36:54.437663 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437670 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" exitCode=0 Jul 29 18:36:54.437663 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437675 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" exitCode=0 Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437680 2574 generic.go:356] "Generic (PLEG): container finished" podID="ff667192-20d6-4975-96ca-777b28d5c25a" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" exitCode=0 Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437709 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437724 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437866 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437912 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437943 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.437979 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.438018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.438059 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} Jul 29 18:36:54.438188 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.438102 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"ff667192-20d6-4975-96ca-777b28d5c25a","Type":"ContainerDied","Data":"7da88407f7de26274e020acefe9912009bda9ea5c3c991a71677e082081d3a21"} Jul 29 18:36:54.445843 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.445809 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.453160 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.453138 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.460275 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.460249 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.462835 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.462809 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:54.466651 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.466628 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:54.469232 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.469218 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.476137 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.476120 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.483485 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.483465 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.490581 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.490561 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.490875 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.490850 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.491016 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.490887 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} err="failed to get container status \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" Jul 29 18:36:54.491016 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.490963 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.491227 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.491207 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.491263 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.491236 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} err="failed to get container status \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" Jul 29 18:36:54.491263 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.491260 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.491481 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.491466 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.491524 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.491488 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} err="failed to get container status \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" Jul 29 18:36:54.491524 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.491506 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.491693 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.491679 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.491731 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.491697 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} err="failed to get container status \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" Jul 29 18:36:54.491731 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.491711 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.491971 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.491904 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.492037 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492007 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} err="failed to get container status \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" Jul 29 18:36:54.492037 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492021 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.492267 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.492249 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.492332 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492274 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} err="failed to get container status \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" Jul 29 18:36:54.492332 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492294 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.492565 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:36:54.492549 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.492615 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492569 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} err="failed to get container status \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" Jul 29 18:36:54.492615 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492582 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.492798 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492779 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} err="failed to get container status \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" Jul 29 18:36:54.492940 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.492800 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.493244 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.493198 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} err="failed to get container status \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" Jul 29 18:36:54.493244 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.493228 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.493593 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.493541 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} err="failed to get container status \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" Jul 29 18:36:54.493593 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.493567 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.494111 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494088 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} err="failed to get container status \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" Jul 29 18:36:54.494205 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494112 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.494416 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494396 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} err="failed to get container status \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" Jul 29 18:36:54.494499 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494418 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.494687 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494666 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} err="failed to get container status \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" Jul 29 18:36:54.494777 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494688 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.494936 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494892 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} err="failed to get container status \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" Jul 29 18:36:54.494985 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.494940 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.495056 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495040 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:54.495224 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495205 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} err="failed to get container status \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" Jul 29 18:36:54.495272 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495228 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.495339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495322 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="init-config-reloader" Jul 29 18:36:54.495388 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495343 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="init-config-reloader" Jul 29 18:36:54.495388 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495358 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-thanos" Jul 29 18:36:54.495388 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495369 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-thanos" Jul 29 18:36:54.495388 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495379 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="config-reloader" Jul 29 18:36:54.495388 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495388 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="config-reloader" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495398 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-web" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495407 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-web" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495419 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="thanos-sidecar" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495425 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="thanos-sidecar" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495433 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="alertmanager" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495430 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} err="failed to get container status \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495446 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495438 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="alertmanager" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495500 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-metric" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495508 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-metric" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495516 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="init-config-reloader" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495523 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="init-config-reloader" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495533 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495540 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495546 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="prom-label-proxy" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495553 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="prom-label-proxy" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495560 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="prometheus" Jul 29 18:36:54.495557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495566 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="prometheus" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495571 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="config-reloader" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495576 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="config-reloader" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495584 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495589 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495603 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-web" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495611 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-web" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495671 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="prom-label-proxy" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495677 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="thanos-sidecar" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495688 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="config-reloader" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495697 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="alertmanager" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495703 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-web" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495707 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy-metric" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495714 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495721 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-thanos" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495727 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="prometheus" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495731 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="config-reloader" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495736 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="4763c8aa-6db3-4cbe-b46a-7c4ed8480e87" containerName="kube-rbac-proxy" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495740 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" containerName="kube-rbac-proxy-web" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495736 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} err="failed to get container status \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495761 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.495989 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} err="failed to get container status \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496011 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496241 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} err="failed to get container status \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" Jul 29 18:36:54.496277 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496264 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.497100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496519 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} err="failed to get container status \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" Jul 29 18:36:54.497100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496536 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.497100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496745 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} err="failed to get container status \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" Jul 29 18:36:54.497100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496767 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.497100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.496998 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} err="failed to get container status \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" Jul 29 18:36:54.497100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497013 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.497365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497169 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} err="failed to get container status \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" Jul 29 18:36:54.497365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497189 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.497456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497435 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} err="failed to get container status \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" Jul 29 18:36:54.497456 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497451 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.497706 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497686 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} err="failed to get container status \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" Jul 29 18:36:54.497706 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497706 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.497951 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497934 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} err="failed to get container status \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" Jul 29 18:36:54.497951 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.497951 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.498171 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498153 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} err="failed to get container status \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" Jul 29 18:36:54.498216 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498173 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.498377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498359 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} err="failed to get container status \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" Jul 29 18:36:54.498446 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498379 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.498594 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498576 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} err="failed to get container status \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" Jul 29 18:36:54.498636 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498595 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.498762 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498747 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} err="failed to get container status \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" Jul 29 18:36:54.498817 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498764 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.498975 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498957 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} err="failed to get container status \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" Jul 29 18:36:54.499051 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.498978 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.499182 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499167 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} err="failed to get container status \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" Jul 29 18:36:54.499230 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499182 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.499443 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499425 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} err="failed to get container status \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" Jul 29 18:36:54.499505 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499444 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.499677 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499657 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} err="failed to get container status \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" Jul 29 18:36:54.499721 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499679 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.499890 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499871 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} err="failed to get container status \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" Jul 29 18:36:54.499957 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.499892 2574 scope.go:117] "RemoveContainer" containerID="91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6" Jul 29 18:36:54.500125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500107 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6"} err="failed to get container status \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": rpc error: code = NotFound desc = could not find container \"91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6\": container with ID starting with 91b210a530a3e47ace4fab879ecf098fc0dda8def874223c34e95bc11acb74e6 not found: ID does not exist" Jul 29 18:36:54.500125 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500125 2574 scope.go:117] "RemoveContainer" containerID="a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b" Jul 29 18:36:54.500212 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500193 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.500296 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500280 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b"} err="failed to get container status \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": rpc error: code = NotFound desc = could not find container \"a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b\": container with ID starting with a944d06d786e024bf1ae666d305e642046b719b79498711dd0fecc56fdde934b not found: ID does not exist" Jul 29 18:36:54.500366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500297 2574 scope.go:117] "RemoveContainer" containerID="e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42" Jul 29 18:36:54.500659 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500512 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42"} err="failed to get container status \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": rpc error: code = NotFound desc = could not find container \"e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42\": container with ID starting with e9ce9e02dd5a51bae0eb999377f975d81738ed8d30e378be8c02716949be1b42 not found: ID does not exist" Jul 29 18:36:54.500659 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500541 2574 scope.go:117] "RemoveContainer" containerID="75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490" Jul 29 18:36:54.500877 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500857 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490"} err="failed to get container status \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": rpc error: code = NotFound desc = could not find container \"75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490\": container with ID starting with 75c243d68f683911f379d0904c5312f7e3895e81e872455b495083dacfdd2490 not found: ID does not exist" Jul 29 18:36:54.500976 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.500880 2574 scope.go:117] "RemoveContainer" containerID="0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a" Jul 29 18:36:54.501229 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.501203 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a"} err="failed to get container status \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": rpc error: code = NotFound desc = could not find container \"0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a\": container with ID starting with 0a4c78175162f26180782c6b5b0b5df88bab9284a456ffad607dac3405d8232a not found: ID does not exist" Jul 29 18:36:54.501297 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.501231 2574 scope.go:117] "RemoveContainer" containerID="47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2" Jul 29 18:36:54.501479 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.501453 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2"} err="failed to get container status \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": rpc error: code = NotFound desc = could not find container \"47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2\": container with ID starting with 47f9ff76dc4822f8653a5ce7c2d4bea41d2214b9fdbbe5a49a115a1a2ee17ae2 not found: ID does not exist" Jul 29 18:36:54.501479 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.501476 2574 scope.go:117] "RemoveContainer" containerID="9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b" Jul 29 18:36:54.501716 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.501697 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b"} err="failed to get container status \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": rpc error: code = NotFound desc = could not find container \"9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b\": container with ID starting with 9fda868b5fe319472bba75c0e6551ce766821126d54d3d335587f3ccb89d458b not found: ID does not exist" Jul 29 18:36:54.503024 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503008 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 29 18:36:54.503209 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503189 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 29 18:36:54.503302 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503191 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 29 18:36:54.503302 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503256 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 29 18:36:54.503479 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503465 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lz7cg\"" Jul 29 18:36:54.503538 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503519 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 29 18:36:54.503635 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503618 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 29 18:36:54.503678 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503647 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 29 18:36:54.503720 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503712 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 29 18:36:54.503906 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503890 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 29 18:36:54.504000 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503971 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-fj2mgd5otp9tp\"" Jul 29 18:36:54.504039 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.504007 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 29 18:36:54.504101 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.504087 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 29 18:36:54.504101 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.503892 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 29 18:36:54.504314 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.504299 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 29 18:36:54.507290 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.507273 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 29 18:36:54.511577 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.511558 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 29 18:36:54.519250 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.519221 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:54.591502 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591469 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591502 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591508 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591702 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591535 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b233f2b3-97d3-43b4-b28c-73bfa117757a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591702 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591603 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591702 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591631 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591702 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591670 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591702 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591702 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591718 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591789 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-config\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-web-config\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwkph\" (UniqueName: \"kubernetes.io/projected/b233f2b3-97d3-43b4-b28c-73bfa117757a-kube-api-access-qwkph\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591848 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.591884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591876 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.592156 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.592156 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.592156 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b233f2b3-97d3-43b4-b28c-73bfa117757a-config-out\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.592156 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.591972 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.692910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.692820 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.692910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.692865 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b233f2b3-97d3-43b4-b28c-73bfa117757a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.692910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.692891 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.692910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.692908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.692975 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.692990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693007 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693022 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693071 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693102 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-config\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693123 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-web-config\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwkph\" (UniqueName: \"kubernetes.io/projected/b233f2b3-97d3-43b4-b28c-73bfa117757a-kube-api-access-qwkph\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693280 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b233f2b3-97d3-43b4-b28c-73bfa117757a-config-out\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.693880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.693567 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.695237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.694247 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.695237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.694664 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.695237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.694897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.696130 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.696070 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b233f2b3-97d3-43b4-b28c-73bfa117757a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.696952 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.696405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.696952 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.696491 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.696952 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.696509 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.696952 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.696568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.696952 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.696719 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.697561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.697296 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-config\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.697561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.697341 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.697561 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.697385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.697956 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.697903 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.698263 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.698243 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.698659 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.698642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b233f2b3-97d3-43b4-b28c-73bfa117757a-config-out\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.698716 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.698652 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-web-config\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.699868 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.699847 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b233f2b3-97d3-43b4-b28c-73bfa117757a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.700449 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.700433 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b233f2b3-97d3-43b4-b28c-73bfa117757a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.703155 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.703136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwkph\" (UniqueName: \"kubernetes.io/projected/b233f2b3-97d3-43b4-b28c-73bfa117757a-kube-api-access-qwkph\") pod \"prometheus-k8s-0\" (UID: \"b233f2b3-97d3-43b4-b28c-73bfa117757a\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.771072 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.771038 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff667192-20d6-4975-96ca-777b28d5c25a" path="/var/lib/kubelet/pods/ff667192-20d6-4975-96ca-777b28d5c25a/volumes" Jul 29 18:36:54.811120 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.811085 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:36:54.948621 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:54.948525 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 29 18:36:54.953156 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:36:54.953132 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb233f2b3_97d3_43b4_b28c_73bfa117757a.slice/crio-1e34c5745703008b66dcfcba0c3da09791d877ed1d54a4cc2227eb5b109e7def WatchSource:0}: Error finding container 1e34c5745703008b66dcfcba0c3da09791d877ed1d54a4cc2227eb5b109e7def: Status 404 returned error can't find the container with id 1e34c5745703008b66dcfcba0c3da09791d877ed1d54a4cc2227eb5b109e7def Jul 29 18:36:55.443044 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:55.443005 2574 generic.go:356] "Generic (PLEG): container finished" podID="b233f2b3-97d3-43b4-b28c-73bfa117757a" containerID="1ea4c7b72eda500a9f3fdda798257db007f896a4779efbc8e18288e034774565" exitCode=0 Jul 29 18:36:55.443210 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:55.443100 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerDied","Data":"1ea4c7b72eda500a9f3fdda798257db007f896a4779efbc8e18288e034774565"} Jul 29 18:36:55.443210 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:55.443191 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"1e34c5745703008b66dcfcba0c3da09791d877ed1d54a4cc2227eb5b109e7def"} Jul 29 18:36:56.454080 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.454040 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"dc726b64131fa36c889abf51b2b49c9616772a151f08ec8976f7145c5196def1"} Jul 29 18:36:56.454569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.454106 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"1a69565e10597afa0ed50b500aa9b5ebdae526e040618a5247838678f5568b04"} Jul 29 18:36:56.454569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.454133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"1fe59f3273807bbf7583198e131a028166a1261f775f4e65af258c771b7423c6"} Jul 29 18:36:56.454569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.454157 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"31d410619ec67768241256090735a4cb42e032bdf62665b28a7391ebba4cb247"} Jul 29 18:36:56.454569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.454181 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"c4e0cab516bb557cdeebc24914d6f1b712962b5b9d6c14cc815ff3054a703ae6"} Jul 29 18:36:56.454569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.454203 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b233f2b3-97d3-43b4-b28c-73bfa117757a","Type":"ContainerStarted","Data":"dd62af7a65844ac2cb7a25dd78a1ac8cc951ff96d69236820047665449494f32"} Jul 29 18:36:56.484599 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:56.484519 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.465893526 podStartE2EDuration="2.484492757s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.599231ms" isStatefulPod=false podCreationTimestamp="2026-07-29 18:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:36:56.482198646 +0000 UTC m=+156.195110136" watchObservedRunningTime="2026-07-29 18:36:56.484492757 +0000 UTC m=+156.197404247" Jul 29 18:36:59.811290 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:36:59.811256 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:09.206884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:09.206839 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778b69bf4c-2l4bf"] Jul 29 18:37:34.227884 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.227816 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-778b69bf4c-2l4bf" podUID="a7f1d4b8-742f-444b-a4ef-dc368debab5a" containerName="console" containerID="cri-o://ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480" gracePeriod=15 Jul 29 18:37:34.481362 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.481303 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:37:34.589521 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.589475 2574 generic.go:356] "Generic (PLEG): container finished" podID="a7f1d4b8-742f-444b-a4ef-dc368debab5a" containerID="ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480" exitCode=0 Jul 29 18:37:34.590028 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.589530 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-778b69bf4c-2l4bf" Jul 29 18:37:34.590028 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.589569 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b69bf4c-2l4bf" event={"ID":"a7f1d4b8-742f-444b-a4ef-dc368debab5a","Type":"ContainerDied","Data":"ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480"} Jul 29 18:37:34.590028 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.589636 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-778b69bf4c-2l4bf" event={"ID":"a7f1d4b8-742f-444b-a4ef-dc368debab5a","Type":"ContainerDied","Data":"bfba223c9767f6c87212cacb0441c72346241c1d746c9928b88ede96a2d16f08"} Jul 29 18:37:34.590028 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.589663 2574 scope.go:117] "RemoveContainer" containerID="ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480" Jul 29 18:37:34.597486 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.597467 2574 scope.go:117] "RemoveContainer" containerID="ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480" Jul 29 18:37:34.597739 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:37:34.597723 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480\": container with ID starting with ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480 not found: ID does not exist" containerID="ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480" Jul 29 18:37:34.597791 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.597746 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480"} err="failed to get container status \"ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480\": rpc error: code = NotFound desc = could not find container \"ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480\": container with ID starting with ddfad9834fc2559ec8ec332c04f0449192d3ca88781466b43c95397c7fa7b480 not found: ID does not exist" Jul 29 18:37:34.612534 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612514 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-trusted-ca-bundle\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.612649 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612547 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-service-ca\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.612649 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612576 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zgmm\" (UniqueName: \"kubernetes.io/projected/a7f1d4b8-742f-444b-a4ef-dc368debab5a-kube-api-access-5zgmm\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.612649 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612605 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-oauth-config\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.612794 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612653 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-oauth-serving-cert\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.612794 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612721 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-config\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.612794 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612752 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-serving-cert\") pod \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\" (UID: \"a7f1d4b8-742f-444b-a4ef-dc368debab5a\") " Jul 29 18:37:34.613075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612950 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:34.613075 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.612969 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:34.613248 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.613148 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:34.613248 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.613166 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-config" (OuterVolumeSpecName: "console-config") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:37:34.613248 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.613233 2574 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-trusted-ca-bundle\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.613366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.613253 2574 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.613366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.613267 2574 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-oauth-serving-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.615110 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.615086 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7f1d4b8-742f-444b-a4ef-dc368debab5a-kube-api-access-5zgmm" (OuterVolumeSpecName: "kube-api-access-5zgmm") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "kube-api-access-5zgmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:37:34.615110 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.615089 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:34.615231 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.615106 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7f1d4b8-742f-444b-a4ef-dc368debab5a" (UID: "a7f1d4b8-742f-444b-a4ef-dc368debab5a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:37:34.714543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.714502 2574 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-config\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.714543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.714538 2574 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-serving-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.714543 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.714548 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zgmm\" (UniqueName: \"kubernetes.io/projected/a7f1d4b8-742f-444b-a4ef-dc368debab5a-kube-api-access-5zgmm\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.714771 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.714557 2574 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7f1d4b8-742f-444b-a4ef-dc368debab5a-console-oauth-config\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:37:34.905847 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.905813 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-778b69bf4c-2l4bf"] Jul 29 18:37:34.909811 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:34.909788 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-778b69bf4c-2l4bf"] Jul 29 18:37:36.765951 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:36.765898 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7f1d4b8-742f-444b-a4ef-dc368debab5a" path="/var/lib/kubelet/pods/a7f1d4b8-742f-444b-a4ef-dc368debab5a/volumes" Jul 29 18:37:54.811510 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:54.811467 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:54.827542 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:54.827510 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:37:55.675223 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:37:55.675193 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 29 18:39:20.620025 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:20.619989 2574 kubelet.go:1648] "Image garbage collection succeeded" Jul 29 18:39:58.758752 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.758708 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld"] Jul 29 18:39:58.761282 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.759129 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7f1d4b8-742f-444b-a4ef-dc368debab5a" containerName="console" Jul 29 18:39:58.761282 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.759148 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7f1d4b8-742f-444b-a4ef-dc368debab5a" containerName="console" Jul 29 18:39:58.761282 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.759207 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7f1d4b8-742f-444b-a4ef-dc368debab5a" containerName="console" Jul 29 18:39:58.762205 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.762180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:58.765584 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.765559 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 29 18:39:58.765700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.765593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-824p8\"" Jul 29 18:39:58.765700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.765559 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 29 18:39:58.776464 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.776438 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld"] Jul 29 18:39:58.884354 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.884313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2mln\" (UniqueName: \"kubernetes.io/projected/97a86d73-1fb1-42f4-88c7-055fb582c419-kube-api-access-p2mln\") pod \"openshift-lws-operator-fd8ccff4c-fk8ld\" (UID: \"97a86d73-1fb1-42f4-88c7-055fb582c419\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:58.884536 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.884368 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97a86d73-1fb1-42f4-88c7-055fb582c419-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fk8ld\" (UID: \"97a86d73-1fb1-42f4-88c7-055fb582c419\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:58.985068 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.985030 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97a86d73-1fb1-42f4-88c7-055fb582c419-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fk8ld\" (UID: \"97a86d73-1fb1-42f4-88c7-055fb582c419\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:58.985258 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.985096 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2mln\" (UniqueName: \"kubernetes.io/projected/97a86d73-1fb1-42f4-88c7-055fb582c419-kube-api-access-p2mln\") pod \"openshift-lws-operator-fd8ccff4c-fk8ld\" (UID: \"97a86d73-1fb1-42f4-88c7-055fb582c419\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:58.985459 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.985439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97a86d73-1fb1-42f4-88c7-055fb582c419-tmp\") pod \"openshift-lws-operator-fd8ccff4c-fk8ld\" (UID: \"97a86d73-1fb1-42f4-88c7-055fb582c419\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:58.995889 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:58.995859 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2mln\" (UniqueName: \"kubernetes.io/projected/97a86d73-1fb1-42f4-88c7-055fb582c419-kube-api-access-p2mln\") pod \"openshift-lws-operator-fd8ccff4c-fk8ld\" (UID: \"97a86d73-1fb1-42f4-88c7-055fb582c419\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:59.087339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:59.087244 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" Jul 29 18:39:59.215850 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:59.215820 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld"] Jul 29 18:39:59.219339 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:39:59.219296 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97a86d73_1fb1_42f4_88c7_055fb582c419.slice/crio-a2870109c53caa2ee4b4fc3db00461b71a0596b699bc4c92976e4e271f11d0c7 WatchSource:0}: Error finding container a2870109c53caa2ee4b4fc3db00461b71a0596b699bc4c92976e4e271f11d0c7: Status 404 returned error can't find the container with id a2870109c53caa2ee4b4fc3db00461b71a0596b699bc4c92976e4e271f11d0c7 Jul 29 18:39:59.220856 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:39:59.220838 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:40:00.079727 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:00.079691 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" event={"ID":"97a86d73-1fb1-42f4-88c7-055fb582c419","Type":"ContainerStarted","Data":"a2870109c53caa2ee4b4fc3db00461b71a0596b699bc4c92976e4e271f11d0c7"} Jul 29 18:40:02.087953 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:02.087901 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" event={"ID":"97a86d73-1fb1-42f4-88c7-055fb582c419","Type":"ContainerStarted","Data":"6cdc0c0d1fda47d6499a2b2f1b58319af6cc726cfdd488b01ce8a0ef70ec384b"} Jul 29 18:40:02.108657 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:02.108424 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-fk8ld" podStartSLOduration=1.4887305610000001 podStartE2EDuration="4.108405013s" totalImagesPullingTime="2.619674452s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:39:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:02.107602761 +0000 UTC m=+341.820514251" watchObservedRunningTime="2026-07-29 18:40:02.108405013 +0000 UTC m=+341.821316506" Jul 29 18:40:17.964834 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.964757 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp"] Jul 29 18:40:17.967807 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.967792 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:17.973325 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.973304 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 29 18:40:17.973615 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.973598 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 29 18:40:17.974045 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.974026 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 29 18:40:17.974141 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.974044 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-b6vgb\"" Jul 29 18:40:17.974428 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:17.974412 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 29 18:40:18.001810 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.001782 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp"] Jul 29 18:40:18.137810 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.137779 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4496f32-73ba-4e03-ba5f-92c628428d13-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.137810 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.137811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4496f32-73ba-4e03-ba5f-92c628428d13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.138062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.137835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmvs9\" (UniqueName: \"kubernetes.io/projected/c4496f32-73ba-4e03-ba5f-92c628428d13-kube-api-access-jmvs9\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.138062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.137908 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4496f32-73ba-4e03-ba5f-92c628428d13-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.138062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.137970 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4496f32-73ba-4e03-ba5f-92c628428d13-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.239206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.239111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4496f32-73ba-4e03-ba5f-92c628428d13-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.239206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.239147 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4496f32-73ba-4e03-ba5f-92c628428d13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.239206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.239170 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmvs9\" (UniqueName: \"kubernetes.io/projected/c4496f32-73ba-4e03-ba5f-92c628428d13-kube-api-access-jmvs9\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.239485 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.239308 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4496f32-73ba-4e03-ba5f-92c628428d13-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.239485 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.239355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4496f32-73ba-4e03-ba5f-92c628428d13-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.239743 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.239723 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/c4496f32-73ba-4e03-ba5f-92c628428d13-manifests\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.241700 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.241671 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c4496f32-73ba-4e03-ba5f-92c628428d13-tmp\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.241805 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.241755 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/c4496f32-73ba-4e03-ba5f-92c628428d13-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.241805 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.241757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/c4496f32-73ba-4e03-ba5f-92c628428d13-webhook-cert\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.254469 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.254439 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmvs9\" (UniqueName: \"kubernetes.io/projected/c4496f32-73ba-4e03-ba5f-92c628428d13-kube-api-access-jmvs9\") pod \"opendatahub-operator-controller-manager-76c6f8f9b-296sp\" (UID: \"c4496f32-73ba-4e03-ba5f-92c628428d13\") " pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.278440 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.278408 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:18.462311 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:18.462275 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp"] Jul 29 18:40:18.465274 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:40:18.465251 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4496f32_73ba_4e03_ba5f_92c628428d13.slice/crio-45712b2b15d5dad8c6db5360d9a12d70e74ddaae4694fbf6e26d9f5c91117ccd WatchSource:0}: Error finding container 45712b2b15d5dad8c6db5360d9a12d70e74ddaae4694fbf6e26d9f5c91117ccd: Status 404 returned error can't find the container with id 45712b2b15d5dad8c6db5360d9a12d70e74ddaae4694fbf6e26d9f5c91117ccd Jul 29 18:40:19.156459 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:19.156414 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" event={"ID":"c4496f32-73ba-4e03-ba5f-92c628428d13","Type":"ContainerStarted","Data":"45712b2b15d5dad8c6db5360d9a12d70e74ddaae4694fbf6e26d9f5c91117ccd"} Jul 29 18:40:22.170801 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:22.170754 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" event={"ID":"c4496f32-73ba-4e03-ba5f-92c628428d13","Type":"ContainerStarted","Data":"e7873ec5b5402d663eda4ffd073c6dd44e4e8b23aebaa07e1946ec6861d27d98"} Jul 29 18:40:23.175505 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:23.175469 2574 generic.go:356] "Generic (PLEG): container finished" podID="c4496f32-73ba-4e03-ba5f-92c628428d13" containerID="e7873ec5b5402d663eda4ffd073c6dd44e4e8b23aebaa07e1946ec6861d27d98" exitCode=0 Jul 29 18:40:23.175886 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:23.175536 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" event={"ID":"c4496f32-73ba-4e03-ba5f-92c628428d13","Type":"ContainerDied","Data":"e7873ec5b5402d663eda4ffd073c6dd44e4e8b23aebaa07e1946ec6861d27d98"} Jul 29 18:40:24.180756 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:24.180719 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" event={"ID":"c4496f32-73ba-4e03-ba5f-92c628428d13","Type":"ContainerStarted","Data":"f4d91fd185eed8d9cab6e43f241fcfd620499c739e2db8c8d1e330c02ecd0439"} Jul 29 18:40:24.181249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:24.180962 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:24.240906 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:24.240857 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" podStartSLOduration=2.571128054 podStartE2EDuration="7.240831819s" totalImagesPullingTime="3.580067388s" totalInitContainerRuntime="1.089636377s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:24.240815643 +0000 UTC m=+363.953727133" watchObservedRunningTime="2026-07-29 18:40:24.240831819 +0000 UTC m=+363.953743307" Jul 29 18:40:35.186613 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:35.186583 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-76c6f8f9b-296sp" Jul 29 18:40:37.620831 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.620796 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98"] Jul 29 18:40:37.757500 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.757465 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98"] Jul 29 18:40:37.757656 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.757558 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.760424 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.760403 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 29 18:40:37.761640 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.761622 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 29 18:40:37.761640 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.761633 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9jtch\"" Jul 29 18:40:37.761852 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.761638 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 29 18:40:37.761852 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.761657 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 29 18:40:37.791666 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.791642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndgr9\" (UniqueName: \"kubernetes.io/projected/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-kube-api-access-ndgr9\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.791766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.791679 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-tmp\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.791766 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.791699 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-tls-certs\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.892487 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.892407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-tmp\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.892487 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.892443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-tls-certs\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.892693 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.892498 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndgr9\" (UniqueName: \"kubernetes.io/projected/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-kube-api-access-ndgr9\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.894821 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.894790 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-tmp\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.895046 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.895030 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-tls-certs\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:37.902042 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:37.902019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndgr9\" (UniqueName: \"kubernetes.io/projected/f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6-kube-api-access-ndgr9\") pod \"kube-auth-proxy-54cd75bc79-tpd98\" (UID: \"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6\") " pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:38.067593 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.067561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" Jul 29 18:40:38.201568 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.201493 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98"] Jul 29 18:40:38.204080 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:40:38.204047 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5bf760c_11ea_46c1_8ad1_43bbe8bc83e6.slice/crio-83948c76a93e04d444cc4f7760f52fccaa47087369afb5be7a7d0c6f9a0d9d12 WatchSource:0}: Error finding container 83948c76a93e04d444cc4f7760f52fccaa47087369afb5be7a7d0c6f9a0d9d12: Status 404 returned error can't find the container with id 83948c76a93e04d444cc4f7760f52fccaa47087369afb5be7a7d0c6f9a0d9d12 Jul 29 18:40:38.231794 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.231763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" event={"ID":"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6","Type":"ContainerStarted","Data":"83948c76a93e04d444cc4f7760f52fccaa47087369afb5be7a7d0c6f9a0d9d12"} Jul 29 18:40:38.784209 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.784179 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sk5nz"] Jul 29 18:40:38.786508 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.786487 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:38.789201 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.789177 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 29 18:40:38.789470 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.789455 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-dnz99\"" Jul 29 18:40:38.797855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.797827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khq8j\" (UniqueName: \"kubernetes.io/projected/8228c2e6-58e9-40a1-8f97-4111fd04b661-kube-api-access-khq8j\") pod \"model-serving-api-54d89bd5fc-sk5nz\" (UID: \"8228c2e6-58e9-40a1-8f97-4111fd04b661\") " pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:38.798004 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.797886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8228c2e6-58e9-40a1-8f97-4111fd04b661-tls-certs\") pod \"model-serving-api-54d89bd5fc-sk5nz\" (UID: \"8228c2e6-58e9-40a1-8f97-4111fd04b661\") " pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:38.799122 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.799094 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sk5nz"] Jul 29 18:40:38.898516 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.898485 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8228c2e6-58e9-40a1-8f97-4111fd04b661-tls-certs\") pod \"model-serving-api-54d89bd5fc-sk5nz\" (UID: \"8228c2e6-58e9-40a1-8f97-4111fd04b661\") " pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:38.898677 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.898588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khq8j\" (UniqueName: \"kubernetes.io/projected/8228c2e6-58e9-40a1-8f97-4111fd04b661-kube-api-access-khq8j\") pod \"model-serving-api-54d89bd5fc-sk5nz\" (UID: \"8228c2e6-58e9-40a1-8f97-4111fd04b661\") " pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:38.901592 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.901568 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8228c2e6-58e9-40a1-8f97-4111fd04b661-tls-certs\") pod \"model-serving-api-54d89bd5fc-sk5nz\" (UID: \"8228c2e6-58e9-40a1-8f97-4111fd04b661\") " pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:38.907104 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:38.907079 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-khq8j\" (UniqueName: \"kubernetes.io/projected/8228c2e6-58e9-40a1-8f97-4111fd04b661-kube-api-access-khq8j\") pod \"model-serving-api-54d89bd5fc-sk5nz\" (UID: \"8228c2e6-58e9-40a1-8f97-4111fd04b661\") " pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:39.077759 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.077642 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q"] Jul 29 18:40:39.080121 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.080100 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.082775 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.082749 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 29 18:40:39.082910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.082850 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 29 18:40:39.083217 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.082984 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 29 18:40:39.083217 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.082998 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 29 18:40:39.083217 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.083027 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-jv89g\"" Jul 29 18:40:39.095011 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.094988 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q"] Jul 29 18:40:39.099885 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.099864 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:39.201433 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201399 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.201591 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201462 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.201591 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.201591 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201538 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.201770 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201612 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.201770 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201653 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.201770 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.201739 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdc4z\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-kube-api-access-zdc4z\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.239729 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.239704 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-sk5nz"] Jul 29 18:40:39.241459 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:40:39.241430 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8228c2e6_58e9_40a1_8f97_4111fd04b661.slice/crio-3409ed72f088a12a7c72e0c768d6ae4de8b4fe0f397b654cde24a336384fdc5d WatchSource:0}: Error finding container 3409ed72f088a12a7c72e0c768d6ae4de8b4fe0f397b654cde24a336384fdc5d: Status 404 returned error can't find the container with id 3409ed72f088a12a7c72e0c768d6ae4de8b4fe0f397b654cde24a336384fdc5d Jul 29 18:40:39.302230 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.302368 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdc4z\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-kube-api-access-zdc4z\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.302368 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302283 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.302368 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.302368 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.302368 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302355 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.302611 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.303144 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.302980 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.304767 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.304742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.304846 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.304830 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.304936 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.304896 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.304978 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.304954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.316639 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.316619 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdc4z\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-kube-api-access-zdc4z\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.319094 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.319076 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-thl4q\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.392189 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.392101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:39.530550 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:39.530528 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q"] Jul 29 18:40:39.532461 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:40:39.532436 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda30c6a_1884_41a6_ac6e_afcdd4da0cff.slice/crio-1c64c4f49cb320ef7fc28fedf5d5808e5714c6fd24cf229452892b9b18721dbf WatchSource:0}: Error finding container 1c64c4f49cb320ef7fc28fedf5d5808e5714c6fd24cf229452892b9b18721dbf: Status 404 returned error can't find the container with id 1c64c4f49cb320ef7fc28fedf5d5808e5714c6fd24cf229452892b9b18721dbf Jul 29 18:40:40.242314 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:40.242273 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" event={"ID":"bda30c6a-1884-41a6-ac6e-afcdd4da0cff","Type":"ContainerStarted","Data":"1c64c4f49cb320ef7fc28fedf5d5808e5714c6fd24cf229452892b9b18721dbf"} Jul 29 18:40:40.243778 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:40.243743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" event={"ID":"8228c2e6-58e9-40a1-8f97-4111fd04b661","Type":"ContainerStarted","Data":"3409ed72f088a12a7c72e0c768d6ae4de8b4fe0f397b654cde24a336384fdc5d"} Jul 29 18:40:41.251213 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:41.251178 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" event={"ID":"8228c2e6-58e9-40a1-8f97-4111fd04b661","Type":"ContainerStarted","Data":"25c223af5d93dce233a98160868f0ebae2ce5f74441dc7ba97d626992c129d0d"} Jul 29 18:40:41.251671 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:41.251405 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:40:41.253064 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:41.253011 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" event={"ID":"f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6","Type":"ContainerStarted","Data":"7f80963449561177c510378d753c283eabe16787228e071dbc05bd9e15c48e5a"} Jul 29 18:40:41.270826 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:41.270768 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" podStartSLOduration=1.466457818 podStartE2EDuration="3.270747646s" totalImagesPullingTime="1.804289828s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:41.268058259 +0000 UTC m=+380.980969753" watchObservedRunningTime="2026-07-29 18:40:41.270747646 +0000 UTC m=+380.983659139" Jul 29 18:40:41.285529 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:41.285481 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-54cd75bc79-tpd98" podStartSLOduration=1.441043407 podStartE2EDuration="4.285461438s" totalImagesPullingTime="2.844418031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:41.285101005 +0000 UTC m=+380.998012497" watchObservedRunningTime="2026-07-29 18:40:41.285461438 +0000 UTC m=+380.998372930" Jul 29 18:40:42.702420 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:42.702380 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:42.702729 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:42.702450 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:43.262851 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.262818 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" event={"ID":"bda30c6a-1884-41a6-ac6e-afcdd4da0cff","Type":"ContainerStarted","Data":"947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999"} Jul 29 18:40:43.263069 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.263021 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:43.264969 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.264913 2574 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-thl4q container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 18:40:43.265087 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.264994 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" podUID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 18:40:43.283643 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.283586 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" podStartSLOduration=1.115838245 podStartE2EDuration="4.283556431s" totalImagesPullingTime="3.167718186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:43.28332177 +0000 UTC m=+382.996233261" watchObservedRunningTime="2026-07-29 18:40:43.283556431 +0000 UTC m=+382.996467929" Jul 29 18:40:43.915710 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.915678 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk"] Jul 29 18:40:43.923200 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.923177 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.925833 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.925811 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-g76sk\"" Jul 29 18:40:43.931110 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.931084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk"] Jul 29 18:40:43.937903 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.937878 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938044 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.937940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938044 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.937980 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938044 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.938023 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938186 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.938061 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-826bt\" (UniqueName: \"kubernetes.io/projected/4e5b83d0-4223-4bf4-85fd-53884cb90808-kube-api-access-826bt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938186 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.938094 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938186 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.938122 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938287 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.938193 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b83d0-4223-4bf4-85fd-53884cb90808-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:43.938287 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:43.938262 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.038903 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.038868 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.038903 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.038907 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.038973 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b83d0-4223-4bf4-85fd-53884cb90808-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039127 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039176 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039172 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039575 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-826bt\" (UniqueName: \"kubernetes.io/projected/4e5b83d0-4223-4bf4-85fd-53884cb90808-kube-api-access-826bt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039575 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039308 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039703 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039761 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039705 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e5b83d0-4223-4bf4-85fd-53884cb90808-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039811 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039768 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.039860 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.039829 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.041607 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.041580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.041832 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.041811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.048661 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.048632 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e5b83d0-4223-4bf4-85fd-53884cb90808-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.049079 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.049055 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-826bt\" (UniqueName: \"kubernetes.io/projected/4e5b83d0-4223-4bf4-85fd-53884cb90808-kube-api-access-826bt\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk\" (UID: \"4e5b83d0-4223-4bf4-85fd-53884cb90808\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.234993 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.234953 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:44.268518 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.268483 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:44.376022 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:44.375995 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk"] Jul 29 18:40:44.378526 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:40:44.378506 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e5b83d0_4223_4bf4_85fd_53884cb90808.slice/crio-26e83dffc39507792bf439b4aa9aa17f7096bc81b55a50189433917063a8e9a8 WatchSource:0}: Error finding container 26e83dffc39507792bf439b4aa9aa17f7096bc81b55a50189433917063a8e9a8: Status 404 returned error can't find the container with id 26e83dffc39507792bf439b4aa9aa17f7096bc81b55a50189433917063a8e9a8 Jul 29 18:40:45.274425 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.274385 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" event={"ID":"4e5b83d0-4223-4bf4-85fd-53884cb90808","Type":"ContainerStarted","Data":"26e83dffc39507792bf439b4aa9aa17f7096bc81b55a50189433917063a8e9a8"} Jul 29 18:40:45.793707 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.793676 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l"] Jul 29 18:40:45.797195 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.797163 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.812263 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.812235 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l"] Jul 29 18:40:45.851298 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851187 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.851298 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851235 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zhlw\" (UniqueName: \"kubernetes.io/projected/93ee3ec0-391d-4b11-ac70-c5a393d66810-kube-api-access-9zhlw\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.851298 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851273 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.852360 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851377 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.852360 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.852360 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851487 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ee3ec0-391d-4b11-ac70-c5a393d66810-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.852360 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.851530 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952247 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952476 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952313 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ee3ec0-391d-4b11-ac70-c5a393d66810-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952476 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952476 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952476 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952442 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zhlw\" (UniqueName: \"kubernetes.io/projected/93ee3ec0-391d-4b11-ac70-c5a393d66810-kube-api-access-9zhlw\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952686 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952487 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.952686 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.952528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.953788 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.953757 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.955717 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.955695 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.955903 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.955881 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.956165 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.956142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/93ee3ec0-391d-4b11-ac70-c5a393d66810-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.956338 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.956317 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.968512 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.968489 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93ee3ec0-391d-4b11-ac70-c5a393d66810-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:45.974238 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:45.974193 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zhlw\" (UniqueName: \"kubernetes.io/projected/93ee3ec0-391d-4b11-ac70-c5a393d66810-kube-api-access-9zhlw\") pod \"istiod-openshift-gateway-5677ccb49b-g4g8l\" (UID: \"93ee3ec0-391d-4b11-ac70-c5a393d66810\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:46.109642 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:46.109552 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:46.267469 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:46.267432 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l"] Jul 29 18:40:47.141699 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:40:47.141666 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ee3ec0_391d_4b11_ac70_c5a393d66810.slice/crio-42a80d8192bc4504668b6259e3db2cad1de7eaabcd4edf6cdb242ac1afb3c098 WatchSource:0}: Error finding container 42a80d8192bc4504668b6259e3db2cad1de7eaabcd4edf6cdb242ac1afb3c098: Status 404 returned error can't find the container with id 42a80d8192bc4504668b6259e3db2cad1de7eaabcd4edf6cdb242ac1afb3c098 Jul 29 18:40:47.144935 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.144886 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:47.145051 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.144980 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:47.203880 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.203843 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:47.203974 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.203959 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:47.204017 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.203998 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:40:47.286681 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.286646 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" event={"ID":"93ee3ec0-391d-4b11-ac70-c5a393d66810","Type":"ContainerStarted","Data":"add6dfd80f2257058526d24a92ebf3d6cf2c56ed2da815e2e4a7c0d98b52b6aa"} Jul 29 18:40:47.286800 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.286703 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" event={"ID":"93ee3ec0-391d-4b11-ac70-c5a393d66810","Type":"ContainerStarted","Data":"42a80d8192bc4504668b6259e3db2cad1de7eaabcd4edf6cdb242ac1afb3c098"} Jul 29 18:40:47.286800 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.286754 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:47.340566 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:47.340513 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" podStartSLOduration=2.34049579 podStartE2EDuration="2.34049579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:47.334861392 +0000 UTC m=+387.047772887" watchObservedRunningTime="2026-07-29 18:40:47.34049579 +0000 UTC m=+387.053407282" Jul 29 18:40:48.297284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.297253 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" event={"ID":"4e5b83d0-4223-4bf4-85fd-53884cb90808","Type":"ContainerStarted","Data":"1d5045744a72d881d3924bc96dfe808140abc184a18794667fca38c3ffa17dd2"} Jul 29 18:40:48.299266 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.299240 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-g4g8l" Jul 29 18:40:48.339659 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.339548 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" podStartSLOduration=2.516792274 podStartE2EDuration="5.339528651s" totalImagesPullingTime="2.822736377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:40:48.338078072 +0000 UTC m=+388.050989561" watchObservedRunningTime="2026-07-29 18:40:48.339528651 +0000 UTC m=+388.052440143" Jul 29 18:40:48.409882 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.409849 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q"] Jul 29 18:40:48.410150 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.410119 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" podUID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" containerName="discovery" containerID="cri-o://947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999" gracePeriod=30 Jul 29 18:40:48.657752 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.657730 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:48.678154 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678135 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-cacerts\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678242 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678165 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-kubeconfig\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678242 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678197 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-local-certs\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678242 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678218 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-ca-configmap\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678398 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678245 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zdc4z\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-kube-api-access-zdc4z\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678398 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678295 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-token\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678398 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678312 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-dns-cert\") pod \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\" (UID: \"bda30c6a-1884-41a6-ac6e-afcdd4da0cff\") " Jul 29 18:40:48.678654 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.678616 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 18:40:48.680670 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.680639 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-kube-api-access-zdc4z" (OuterVolumeSpecName: "kube-api-access-zdc4z") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "kube-api-access-zdc4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:40:48.680756 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.680676 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-cacerts" (OuterVolumeSpecName: "cacerts") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:40:48.680849 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.680823 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:40:48.680962 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.680867 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-local-certs" (OuterVolumeSpecName: "local-certs") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:40:48.681157 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.681139 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:40:48.681206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.681151 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-token" (OuterVolumeSpecName: "istio-token") pod "bda30c6a-1884-41a6-ac6e-afcdd4da0cff" (UID: "bda30c6a-1884-41a6-ac6e-afcdd4da0cff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:40:48.779415 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779390 2574 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-cacerts\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:48.779415 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779412 2574 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-kubeconfig\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:48.779566 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779422 2574 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-local-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:48.779566 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779430 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-ca-configmap\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:48.779566 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779439 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdc4z\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-kube-api-access-zdc4z\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:48.779566 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779447 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:48.779566 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:48.779455 2574 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/bda30c6a-1884-41a6-ac6e-afcdd4da0cff-istio-csr-dns-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:40:49.235934 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.235895 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:49.240393 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.240370 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:49.302311 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.302278 2574 generic.go:356] "Generic (PLEG): container finished" podID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" containerID="947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999" exitCode=0 Jul 29 18:40:49.302748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.302329 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" Jul 29 18:40:49.302748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.302358 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" event={"ID":"bda30c6a-1884-41a6-ac6e-afcdd4da0cff","Type":"ContainerDied","Data":"947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999"} Jul 29 18:40:49.302748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.302403 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q" event={"ID":"bda30c6a-1884-41a6-ac6e-afcdd4da0cff","Type":"ContainerDied","Data":"1c64c4f49cb320ef7fc28fedf5d5808e5714c6fd24cf229452892b9b18721dbf"} Jul 29 18:40:49.302748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.302422 2574 scope.go:117] "RemoveContainer" containerID="947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999" Jul 29 18:40:49.302748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.302735 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:49.303635 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.303620 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk" Jul 29 18:40:49.310691 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.310675 2574 scope.go:117] "RemoveContainer" containerID="947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999" Jul 29 18:40:49.310967 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:40:49.310949 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999\": container with ID starting with 947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999 not found: ID does not exist" containerID="947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999" Jul 29 18:40:49.311031 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.310974 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999"} err="failed to get container status \"947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999\": rpc error: code = NotFound desc = could not find container \"947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999\": container with ID starting with 947227446df1494f619533eb441d378e56c2903027579afc77ca4a785aee0999 not found: ID does not exist" Jul 29 18:40:49.321032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.321009 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q"] Jul 29 18:40:49.324866 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:49.324847 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-thl4q"] Jul 29 18:40:50.766522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:50.766488 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" path="/var/lib/kubelet/pods/bda30c6a-1884-41a6-ac6e-afcdd4da0cff/volumes" Jul 29 18:40:52.262030 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:40:52.262004 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-sk5nz" Jul 29 18:41:38.875285 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.875251 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rfgq"] Jul 29 18:41:38.875695 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.875531 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" containerName="discovery" Jul 29 18:41:38.875695 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.875542 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" containerName="discovery" Jul 29 18:41:38.875695 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.875593 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="bda30c6a-1884-41a6-ac6e-afcdd4da0cff" containerName="discovery" Jul 29 18:41:38.888167 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.888139 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:38.888708 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.888678 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rfgq"] Jul 29 18:41:38.891114 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.891089 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 29 18:41:38.891243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.891141 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 29 18:41:38.892259 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.892235 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-spkc9\"" Jul 29 18:41:38.991850 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:38.991814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45tf5\" (UniqueName: \"kubernetes.io/projected/6b1b358c-60d0-45db-80cc-48223fccc79e-kube-api-access-45tf5\") pod \"kuadrant-operator-catalog-4rfgq\" (UID: \"6b1b358c-60d0-45db-80cc-48223fccc79e\") " pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:39.093291 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.093260 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45tf5\" (UniqueName: \"kubernetes.io/projected/6b1b358c-60d0-45db-80cc-48223fccc79e-kube-api-access-45tf5\") pod \"kuadrant-operator-catalog-4rfgq\" (UID: \"6b1b358c-60d0-45db-80cc-48223fccc79e\") " pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:39.101684 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.101658 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45tf5\" (UniqueName: \"kubernetes.io/projected/6b1b358c-60d0-45db-80cc-48223fccc79e-kube-api-access-45tf5\") pod \"kuadrant-operator-catalog-4rfgq\" (UID: \"6b1b358c-60d0-45db-80cc-48223fccc79e\") " pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:39.200247 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.200221 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:39.237804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.237769 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rfgq"] Jul 29 18:41:39.332104 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.332077 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rfgq"] Jul 29 18:41:39.334819 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:41:39.334796 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1b358c_60d0_45db_80cc_48223fccc79e.slice/crio-2bb57d3f27789269fe261ed1b283208abda2fdd1b49ed2e7e9c5ddfd4e7d1aa6 WatchSource:0}: Error finding container 2bb57d3f27789269fe261ed1b283208abda2fdd1b49ed2e7e9c5ddfd4e7d1aa6: Status 404 returned error can't find the container with id 2bb57d3f27789269fe261ed1b283208abda2fdd1b49ed2e7e9c5ddfd4e7d1aa6 Jul 29 18:41:39.453876 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.453799 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-trfbj"] Jul 29 18:41:39.458071 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.458056 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:39.464473 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.464444 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-trfbj"] Jul 29 18:41:39.500062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.500024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" event={"ID":"6b1b358c-60d0-45db-80cc-48223fccc79e","Type":"ContainerStarted","Data":"2bb57d3f27789269fe261ed1b283208abda2fdd1b49ed2e7e9c5ddfd4e7d1aa6"} Jul 29 18:41:39.598027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.597986 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djkj2\" (UniqueName: \"kubernetes.io/projected/a69c9973-8278-46f0-ab9d-d9b3b97cd4a3-kube-api-access-djkj2\") pod \"kuadrant-operator-catalog-trfbj\" (UID: \"a69c9973-8278-46f0-ab9d-d9b3b97cd4a3\") " pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:39.698768 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.698739 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djkj2\" (UniqueName: \"kubernetes.io/projected/a69c9973-8278-46f0-ab9d-d9b3b97cd4a3-kube-api-access-djkj2\") pod \"kuadrant-operator-catalog-trfbj\" (UID: \"a69c9973-8278-46f0-ab9d-d9b3b97cd4a3\") " pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:39.707509 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.707443 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djkj2\" (UniqueName: \"kubernetes.io/projected/a69c9973-8278-46f0-ab9d-d9b3b97cd4a3-kube-api-access-djkj2\") pod \"kuadrant-operator-catalog-trfbj\" (UID: \"a69c9973-8278-46f0-ab9d-d9b3b97cd4a3\") " pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:39.768249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.768203 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:39.900217 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:39.900185 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-trfbj"] Jul 29 18:41:39.902703 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:41:39.902660 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda69c9973_8278_46f0_ab9d_d9b3b97cd4a3.slice/crio-f5c7b5c9b9e8c900ade475ceec7c99a14a041b4f305d3891555ca05fc62581d3 WatchSource:0}: Error finding container f5c7b5c9b9e8c900ade475ceec7c99a14a041b4f305d3891555ca05fc62581d3: Status 404 returned error can't find the container with id f5c7b5c9b9e8c900ade475ceec7c99a14a041b4f305d3891555ca05fc62581d3 Jul 29 18:41:40.506080 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:40.506045 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" event={"ID":"a69c9973-8278-46f0-ab9d-d9b3b97cd4a3","Type":"ContainerStarted","Data":"f5c7b5c9b9e8c900ade475ceec7c99a14a041b4f305d3891555ca05fc62581d3"} Jul 29 18:41:42.515592 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.515557 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" event={"ID":"a69c9973-8278-46f0-ab9d-d9b3b97cd4a3","Type":"ContainerStarted","Data":"fcf0bce1bfd994c59c3cc9cf9e44d7d17f1cb9a7d1a95e3bb63cb2eadce2573e"} Jul 29 18:41:42.517144 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.517111 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" event={"ID":"6b1b358c-60d0-45db-80cc-48223fccc79e","Type":"ContainerStarted","Data":"ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551"} Jul 29 18:41:42.517268 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.517157 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" podUID="6b1b358c-60d0-45db-80cc-48223fccc79e" containerName="registry-server" containerID="cri-o://ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551" gracePeriod=2 Jul 29 18:41:42.532596 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.532555 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" podStartSLOduration=1.657040122 podStartE2EDuration="3.532542153s" totalImagesPullingTime="1.875502031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:41:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:41:42.530853773 +0000 UTC m=+442.243765264" watchObservedRunningTime="2026-07-29 18:41:42.532542153 +0000 UTC m=+442.245453675" Jul 29 18:41:42.546964 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.546907 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" podStartSLOduration=2.105354053 podStartE2EDuration="4.546894059s" totalImagesPullingTime="2.441540006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:41:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:41:42.545453054 +0000 UTC m=+442.258364545" watchObservedRunningTime="2026-07-29 18:41:42.546894059 +0000 UTC m=+442.259805547" Jul 29 18:41:42.753463 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.753441 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:42.826500 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.826431 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45tf5\" (UniqueName: \"kubernetes.io/projected/6b1b358c-60d0-45db-80cc-48223fccc79e-kube-api-access-45tf5\") pod \"6b1b358c-60d0-45db-80cc-48223fccc79e\" (UID: \"6b1b358c-60d0-45db-80cc-48223fccc79e\") " Jul 29 18:41:42.828664 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.828634 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1b358c-60d0-45db-80cc-48223fccc79e-kube-api-access-45tf5" (OuterVolumeSpecName: "kube-api-access-45tf5") pod "6b1b358c-60d0-45db-80cc-48223fccc79e" (UID: "6b1b358c-60d0-45db-80cc-48223fccc79e"). InnerVolumeSpecName "kube-api-access-45tf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:41:42.927905 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:42.927878 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45tf5\" (UniqueName: \"kubernetes.io/projected/6b1b358c-60d0-45db-80cc-48223fccc79e-kube-api-access-45tf5\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:41:43.522012 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.521976 2574 generic.go:356] "Generic (PLEG): container finished" podID="6b1b358c-60d0-45db-80cc-48223fccc79e" containerID="ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551" exitCode=0 Jul 29 18:41:43.522405 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.522017 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" Jul 29 18:41:43.522405 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.522067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" event={"ID":"6b1b358c-60d0-45db-80cc-48223fccc79e","Type":"ContainerDied","Data":"ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551"} Jul 29 18:41:43.522405 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.522108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4rfgq" event={"ID":"6b1b358c-60d0-45db-80cc-48223fccc79e","Type":"ContainerDied","Data":"2bb57d3f27789269fe261ed1b283208abda2fdd1b49ed2e7e9c5ddfd4e7d1aa6"} Jul 29 18:41:43.522405 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.522129 2574 scope.go:117] "RemoveContainer" containerID="ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551" Jul 29 18:41:43.531265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.531250 2574 scope.go:117] "RemoveContainer" containerID="ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551" Jul 29 18:41:43.531504 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:41:43.531483 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551\": container with ID starting with ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551 not found: ID does not exist" containerID="ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551" Jul 29 18:41:43.531563 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.531512 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551"} err="failed to get container status \"ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551\": rpc error: code = NotFound desc = could not find container \"ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551\": container with ID starting with ee308bc605a869b95603c1924e69c23a7e7de1c5c73dbfcf2520f30d82e13551 not found: ID does not exist" Jul 29 18:41:43.543213 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.543191 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rfgq"] Jul 29 18:41:43.547476 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:43.547455 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4rfgq"] Jul 29 18:41:44.768194 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:44.768164 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b1b358c-60d0-45db-80cc-48223fccc79e" path="/var/lib/kubelet/pods/6b1b358c-60d0-45db-80cc-48223fccc79e/volumes" Jul 29 18:41:49.768991 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:49.768942 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:49.769412 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:49.769013 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:49.790864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:49.790838 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:41:50.572505 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:41:50.572476 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-trfbj" Jul 29 18:42:00.569802 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.569760 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-689bcc974d-sc47h"] Jul 29 18:42:00.570326 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.570069 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b1b358c-60d0-45db-80cc-48223fccc79e" containerName="registry-server" Jul 29 18:42:00.570326 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.570081 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b1b358c-60d0-45db-80cc-48223fccc79e" containerName="registry-server" Jul 29 18:42:00.570326 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.570137 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b1b358c-60d0-45db-80cc-48223fccc79e" containerName="registry-server" Jul 29 18:42:00.574549 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.574530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.577296 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.577275 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 29 18:42:00.577296 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.577291 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 29 18:42:00.578422 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.578400 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 29 18:42:00.578538 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.578435 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 29 18:42:00.578538 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.578449 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 29 18:42:00.578538 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.578484 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-9hd9l\"" Jul 29 18:42:00.578538 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.578522 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 29 18:42:00.578736 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.578659 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 29 18:42:00.583625 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.583575 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 29 18:42:00.583739 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.583687 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689bcc974d-sc47h"] Jul 29 18:42:00.673380 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-trusted-ca-bundle\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.673549 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673403 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-oauth-config\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.673549 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673453 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-serving-cert\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.673636 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-service-ca\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.673636 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-oauth-serving-cert\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.673740 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673657 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-config\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.673740 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.673708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkxln\" (UniqueName: \"kubernetes.io/projected/c26d0e1e-95a0-484c-9914-f11c3143f61a-kube-api-access-tkxln\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774615 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774590 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-trusted-ca-bundle\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774615 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-oauth-config\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774655 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-serving-cert\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774682 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-service-ca\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774706 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-oauth-serving-cert\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774726 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-config\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.774855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.774744 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkxln\" (UniqueName: \"kubernetes.io/projected/c26d0e1e-95a0-484c-9914-f11c3143f61a-kube-api-access-tkxln\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.775565 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.775539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-config\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.775698 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.775618 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-service-ca\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.775698 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.775682 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-trusted-ca-bundle\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.775817 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.775748 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c26d0e1e-95a0-484c-9914-f11c3143f61a-oauth-serving-cert\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.777635 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.777610 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-serving-cert\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.778168 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.778147 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c26d0e1e-95a0-484c-9914-f11c3143f61a-console-oauth-config\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.786794 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.786762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkxln\" (UniqueName: \"kubernetes.io/projected/c26d0e1e-95a0-484c-9914-f11c3143f61a-kube-api-access-tkxln\") pod \"console-689bcc974d-sc47h\" (UID: \"c26d0e1e-95a0-484c-9914-f11c3143f61a\") " pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:00.885601 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:00.885534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:01.016761 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:01.016728 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-689bcc974d-sc47h"] Jul 29 18:42:01.019509 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:42:01.019484 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26d0e1e_95a0_484c_9914_f11c3143f61a.slice/crio-7f8b56e08cb1975a83cd4dfb3e58a07e12487a69e6a1801a8e2b97d97a9e9cb2 WatchSource:0}: Error finding container 7f8b56e08cb1975a83cd4dfb3e58a07e12487a69e6a1801a8e2b97d97a9e9cb2: Status 404 returned error can't find the container with id 7f8b56e08cb1975a83cd4dfb3e58a07e12487a69e6a1801a8e2b97d97a9e9cb2 Jul 29 18:42:01.594739 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:01.594705 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bcc974d-sc47h" event={"ID":"c26d0e1e-95a0-484c-9914-f11c3143f61a","Type":"ContainerStarted","Data":"9afce9615d2c4201240170474c41e29e6f5e7ca557c6720beff8b2900389b1d5"} Jul 29 18:42:01.595156 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:01.594752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-689bcc974d-sc47h" event={"ID":"c26d0e1e-95a0-484c-9914-f11c3143f61a","Type":"ContainerStarted","Data":"7f8b56e08cb1975a83cd4dfb3e58a07e12487a69e6a1801a8e2b97d97a9e9cb2"} Jul 29 18:42:01.614626 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:01.614581 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-689bcc974d-sc47h" podStartSLOduration=1.614565116 podStartE2EDuration="1.614565116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:01.612976418 +0000 UTC m=+461.325887910" watchObservedRunningTime="2026-07-29 18:42:01.614565116 +0000 UTC m=+461.327476607" Jul 29 18:42:10.886483 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:10.886447 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:10.886868 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:10.886518 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:10.891159 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:10.891137 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:11.639985 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:11.639955 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-689bcc974d-sc47h" Jul 29 18:42:12.526404 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.526373 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66"] Jul 29 18:42:12.529726 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.529710 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.534272 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.534241 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zn4dv\"" Jul 29 18:42:12.552971 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.552951 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66"] Jul 29 18:42:12.676540 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.676506 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6378fb07-dc0a-4993-b1f9-08f083aa0abd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qkd66\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.676698 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.676547 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6z29\" (UniqueName: \"kubernetes.io/projected/6378fb07-dc0a-4993-b1f9-08f083aa0abd-kube-api-access-v6z29\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qkd66\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.778062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.777979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6378fb07-dc0a-4993-b1f9-08f083aa0abd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qkd66\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.778062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.778045 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6z29\" (UniqueName: \"kubernetes.io/projected/6378fb07-dc0a-4993-b1f9-08f083aa0abd-kube-api-access-v6z29\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qkd66\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.778399 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.778359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6378fb07-dc0a-4993-b1f9-08f083aa0abd-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qkd66\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.787281 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.787257 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6z29\" (UniqueName: \"kubernetes.io/projected/6378fb07-dc0a-4993-b1f9-08f083aa0abd-kube-api-access-v6z29\") pod \"kuadrant-operator-controller-manager-8684c9ff49-qkd66\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.839864 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.839838 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:12.973935 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:12.973758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66"] Jul 29 18:42:12.976877 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:42:12.976850 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6378fb07_dc0a_4993_b1f9_08f083aa0abd.slice/crio-dac8a8fbb1711c946cbde671610240e2ffce2573d2d1c636bb34567b8cd62b86 WatchSource:0}: Error finding container dac8a8fbb1711c946cbde671610240e2ffce2573d2d1c636bb34567b8cd62b86: Status 404 returned error can't find the container with id dac8a8fbb1711c946cbde671610240e2ffce2573d2d1c636bb34567b8cd62b86 Jul 29 18:42:13.644987 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:13.644950 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" event={"ID":"6378fb07-dc0a-4993-b1f9-08f083aa0abd","Type":"ContainerStarted","Data":"dac8a8fbb1711c946cbde671610240e2ffce2573d2d1c636bb34567b8cd62b86"} Jul 29 18:42:15.135550 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.135517 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q6spr"] Jul 29 18:42:15.138192 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.138164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:15.140712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.140687 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5x7zx\"" Jul 29 18:42:15.149530 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.149503 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q6spr"] Jul 29 18:42:15.200413 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.200374 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-492qz\" (UniqueName: \"kubernetes.io/projected/bf35b243-3d17-4d07-9f3f-3165aa313ae1-kube-api-access-492qz\") pod \"authorino-operator-75cc8d5cb7-q6spr\" (UID: \"bf35b243-3d17-4d07-9f3f-3165aa313ae1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:15.301142 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.301112 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-492qz\" (UniqueName: \"kubernetes.io/projected/bf35b243-3d17-4d07-9f3f-3165aa313ae1-kube-api-access-492qz\") pod \"authorino-operator-75cc8d5cb7-q6spr\" (UID: \"bf35b243-3d17-4d07-9f3f-3165aa313ae1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:15.310152 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.310128 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-492qz\" (UniqueName: \"kubernetes.io/projected/bf35b243-3d17-4d07-9f3f-3165aa313ae1-kube-api-access-492qz\") pod \"authorino-operator-75cc8d5cb7-q6spr\" (UID: \"bf35b243-3d17-4d07-9f3f-3165aa313ae1\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:15.451310 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.451275 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:15.580504 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.580478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-q6spr"] Jul 29 18:42:15.583183 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:42:15.583161 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf35b243_3d17_4d07_9f3f_3165aa313ae1.slice/crio-1713ff5bf48b8848af0be0b5875f0fdb8ce6bfada7974d5c1e9d258a238b5ce1 WatchSource:0}: Error finding container 1713ff5bf48b8848af0be0b5875f0fdb8ce6bfada7974d5c1e9d258a238b5ce1: Status 404 returned error can't find the container with id 1713ff5bf48b8848af0be0b5875f0fdb8ce6bfada7974d5c1e9d258a238b5ce1 Jul 29 18:42:15.655095 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:15.655061 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" event={"ID":"bf35b243-3d17-4d07-9f3f-3165aa313ae1","Type":"ContainerStarted","Data":"1713ff5bf48b8848af0be0b5875f0fdb8ce6bfada7974d5c1e9d258a238b5ce1"} Jul 29 18:42:19.676252 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:19.676211 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" event={"ID":"bf35b243-3d17-4d07-9f3f-3165aa313ae1","Type":"ContainerStarted","Data":"edb051eeb1ef01833ed2edd01759ae4616b3865313df4bc5214c325b232892dc"} Jul 29 18:42:19.676639 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:19.676302 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:19.677852 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:19.677826 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" event={"ID":"6378fb07-dc0a-4993-b1f9-08f083aa0abd","Type":"ContainerStarted","Data":"d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084"} Jul 29 18:42:19.678012 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:19.677982 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:19.712338 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:19.712296 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" podStartSLOduration=1.1361575 podStartE2EDuration="4.712282344s" totalImagesPullingTime="3.576124844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:19.710438462 +0000 UTC m=+479.423349951" watchObservedRunningTime="2026-07-29 18:42:19.712282344 +0000 UTC m=+479.425193828" Jul 29 18:42:19.742055 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:19.742002 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" podStartSLOduration=1.5512688460000001 podStartE2EDuration="7.741983275s" totalImagesPullingTime="6.190714429s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:19.740574584 +0000 UTC m=+479.453486073" watchObservedRunningTime="2026-07-29 18:42:19.741983275 +0000 UTC m=+479.454894766" Jul 29 18:42:30.686512 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:30.686483 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:30.686974 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:30.686550 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-q6spr" Jul 29 18:42:32.725626 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.725589 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66"] Jul 29 18:42:32.726027 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.725785 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" podUID="6378fb07-dc0a-4993-b1f9-08f083aa0abd" containerName="manager" containerID="cri-o://d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084" gracePeriod=2 Jul 29 18:42:32.735322 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.735070 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66"] Jul 29 18:42:32.767304 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.767283 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk"] Jul 29 18:42:32.767569 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.767558 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6378fb07-dc0a-4993-b1f9-08f083aa0abd" containerName="manager" Jul 29 18:42:32.767610 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.767571 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6378fb07-dc0a-4993-b1f9-08f083aa0abd" containerName="manager" Jul 29 18:42:32.767642 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.767619 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6378fb07-dc0a-4993-b1f9-08f083aa0abd" containerName="manager" Jul 29 18:42:32.769471 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.769451 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:32.788611 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.788590 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk"] Jul 29 18:42:32.959525 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.959491 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ndk5\" (UniqueName: \"kubernetes.io/projected/6e8f0431-6ba4-4f37-bb78-675617e72fdc-kube-api-access-9ndk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q9zvk\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:32.959651 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.959540 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8f0431-6ba4-4f37-bb78-675617e72fdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q9zvk\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:32.975836 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:32.975783 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:33.060841 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.060815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ndk5\" (UniqueName: \"kubernetes.io/projected/6e8f0431-6ba4-4f37-bb78-675617e72fdc-kube-api-access-9ndk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q9zvk\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:33.060989 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.060846 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8f0431-6ba4-4f37-bb78-675617e72fdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q9zvk\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:33.061197 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.061167 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8f0431-6ba4-4f37-bb78-675617e72fdc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q9zvk\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:33.074094 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.074071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ndk5\" (UniqueName: \"kubernetes.io/projected/6e8f0431-6ba4-4f37-bb78-675617e72fdc-kube-api-access-9ndk5\") pod \"kuadrant-operator-controller-manager-8684c9ff49-q9zvk\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:33.100049 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.100025 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:33.162249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.162146 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v6z29\" (UniqueName: \"kubernetes.io/projected/6378fb07-dc0a-4993-b1f9-08f083aa0abd-kube-api-access-v6z29\") pod \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " Jul 29 18:42:33.162249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.162234 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6378fb07-dc0a-4993-b1f9-08f083aa0abd-extensions-socket-volume\") pod \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\" (UID: \"6378fb07-dc0a-4993-b1f9-08f083aa0abd\") " Jul 29 18:42:33.162876 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.162828 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6378fb07-dc0a-4993-b1f9-08f083aa0abd-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6378fb07-dc0a-4993-b1f9-08f083aa0abd" (UID: "6378fb07-dc0a-4993-b1f9-08f083aa0abd"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:42:33.164144 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.164123 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6378fb07-dc0a-4993-b1f9-08f083aa0abd-kube-api-access-v6z29" (OuterVolumeSpecName: "kube-api-access-v6z29") pod "6378fb07-dc0a-4993-b1f9-08f083aa0abd" (UID: "6378fb07-dc0a-4993-b1f9-08f083aa0abd"). InnerVolumeSpecName "kube-api-access-v6z29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:42:33.202460 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.202426 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7"] Jul 29 18:42:33.206413 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.206387 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.219851 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.219829 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7"] Jul 29 18:42:33.240858 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.240830 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk"] Jul 29 18:42:33.243263 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:42:33.243239 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8f0431_6ba4_4f37_bb78_675617e72fdc.slice/crio-d6a1091daa4799797dee9b58442e0ba0b7323eb2ad89f0846d87a12e08c24ab7 WatchSource:0}: Error finding container d6a1091daa4799797dee9b58442e0ba0b7323eb2ad89f0846d87a12e08c24ab7: Status 404 returned error can't find the container with id d6a1091daa4799797dee9b58442e0ba0b7323eb2ad89f0846d87a12e08c24ab7 Jul 29 18:42:33.263671 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.263652 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6z29\" (UniqueName: \"kubernetes.io/projected/6378fb07-dc0a-4993-b1f9-08f083aa0abd-kube-api-access-v6z29\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:42:33.263748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.263673 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6378fb07-dc0a-4993-b1f9-08f083aa0abd-extensions-socket-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:42:33.364804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.364775 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z2jx7\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.364907 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.364833 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4462l\" (UniqueName: \"kubernetes.io/projected/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-kube-api-access-4462l\") pod \"kuadrant-operator-controller-manager-86878c769f-z2jx7\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.465850 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.465816 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z2jx7\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.466009 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.465878 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4462l\" (UniqueName: \"kubernetes.io/projected/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-kube-api-access-4462l\") pod \"kuadrant-operator-controller-manager-86878c769f-z2jx7\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.466206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.466187 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-z2jx7\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.477826 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.477805 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4462l\" (UniqueName: \"kubernetes.io/projected/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-kube-api-access-4462l\") pod \"kuadrant-operator-controller-manager-86878c769f-z2jx7\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.517684 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.517623 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.668776 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.668333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7"] Jul 29 18:42:33.671750 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:42:33.671722 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeee50a3d_e4f6_46a3_b7c7_0236ab7cfd52.slice/crio-1e021c47f0f41ea0fe96fa499cbed0fa704fadb3f9b9338750da6af59a7146f2 WatchSource:0}: Error finding container 1e021c47f0f41ea0fe96fa499cbed0fa704fadb3f9b9338750da6af59a7146f2: Status 404 returned error can't find the container with id 1e021c47f0f41ea0fe96fa499cbed0fa704fadb3f9b9338750da6af59a7146f2 Jul 29 18:42:33.740162 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.740131 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" event={"ID":"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52","Type":"ContainerStarted","Data":"32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828"} Jul 29 18:42:33.740522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.740283 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:33.740522 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.740324 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" event={"ID":"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52","Type":"ContainerStarted","Data":"1e021c47f0f41ea0fe96fa499cbed0fa704fadb3f9b9338750da6af59a7146f2"} Jul 29 18:42:33.741712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.741690 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" event={"ID":"6e8f0431-6ba4-4f37-bb78-675617e72fdc","Type":"ContainerStarted","Data":"2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f"} Jul 29 18:42:33.741817 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.741734 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" event={"ID":"6e8f0431-6ba4-4f37-bb78-675617e72fdc","Type":"ContainerStarted","Data":"d6a1091daa4799797dee9b58442e0ba0b7323eb2ad89f0846d87a12e08c24ab7"} Jul 29 18:42:33.741817 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.741763 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:33.743043 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.743025 2574 generic.go:356] "Generic (PLEG): container finished" podID="6378fb07-dc0a-4993-b1f9-08f083aa0abd" containerID="d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084" exitCode=0 Jul 29 18:42:33.743139 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.743066 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-qkd66" Jul 29 18:42:33.743139 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.743080 2574 scope.go:117] "RemoveContainer" containerID="d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084" Jul 29 18:42:33.754111 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.754090 2574 scope.go:117] "RemoveContainer" containerID="d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084" Jul 29 18:42:33.754343 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:42:33.754325 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084\": container with ID starting with d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084 not found: ID does not exist" containerID="d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084" Jul 29 18:42:33.754404 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.754354 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084"} err="failed to get container status \"d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084\": rpc error: code = NotFound desc = could not find container \"d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084\": container with ID starting with d60e66241b75917015d750bb8048c33d3750c97cabe2d8177cfdd38c65aa1084 not found: ID does not exist" Jul 29 18:42:33.772039 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.771974 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" podStartSLOduration=0.771962057 podStartE2EDuration="771.962057ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:33.770586073 +0000 UTC m=+493.483497563" watchObservedRunningTime="2026-07-29 18:42:33.771962057 +0000 UTC m=+493.484873541" Jul 29 18:42:33.807306 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:33.807259 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" podStartSLOduration=1.807247778 podStartE2EDuration="1.807247778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:42:33.804488298 +0000 UTC m=+493.517399788" watchObservedRunningTime="2026-07-29 18:42:33.807247778 +0000 UTC m=+493.520159267" Jul 29 18:42:34.766809 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:34.766776 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6378fb07-dc0a-4993-b1f9-08f083aa0abd" path="/var/lib/kubelet/pods/6378fb07-dc0a-4993-b1f9-08f083aa0abd/volumes" Jul 29 18:42:44.750648 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:44.750615 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:44.751076 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:44.750804 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:42:44.828525 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:44.828496 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk"] Jul 29 18:42:44.828699 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:44.828678 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" podUID="6e8f0431-6ba4-4f37-bb78-675617e72fdc" containerName="manager" containerID="cri-o://2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f" gracePeriod=10 Jul 29 18:42:45.073614 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.073586 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:45.154986 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.154958 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8f0431-6ba4-4f37-bb78-675617e72fdc-extensions-socket-volume\") pod \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " Jul 29 18:42:45.155180 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.155044 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ndk5\" (UniqueName: \"kubernetes.io/projected/6e8f0431-6ba4-4f37-bb78-675617e72fdc-kube-api-access-9ndk5\") pod \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\" (UID: \"6e8f0431-6ba4-4f37-bb78-675617e72fdc\") " Jul 29 18:42:45.155341 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.155320 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e8f0431-6ba4-4f37-bb78-675617e72fdc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6e8f0431-6ba4-4f37-bb78-675617e72fdc" (UID: "6e8f0431-6ba4-4f37-bb78-675617e72fdc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:42:45.157081 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.157060 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8f0431-6ba4-4f37-bb78-675617e72fdc-kube-api-access-9ndk5" (OuterVolumeSpecName: "kube-api-access-9ndk5") pod "6e8f0431-6ba4-4f37-bb78-675617e72fdc" (UID: "6e8f0431-6ba4-4f37-bb78-675617e72fdc"). InnerVolumeSpecName "kube-api-access-9ndk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:42:45.256584 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.256544 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6e8f0431-6ba4-4f37-bb78-675617e72fdc-extensions-socket-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.256584 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.256575 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ndk5\" (UniqueName: \"kubernetes.io/projected/6e8f0431-6ba4-4f37-bb78-675617e72fdc-kube-api-access-9ndk5\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:42:45.803712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.803376 2574 generic.go:356] "Generic (PLEG): container finished" podID="6e8f0431-6ba4-4f37-bb78-675617e72fdc" containerID="2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f" exitCode=0 Jul 29 18:42:45.803712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.803433 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" event={"ID":"6e8f0431-6ba4-4f37-bb78-675617e72fdc","Type":"ContainerDied","Data":"2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f"} Jul 29 18:42:45.803712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.803500 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" Jul 29 18:42:45.803712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.803523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk" event={"ID":"6e8f0431-6ba4-4f37-bb78-675617e72fdc","Type":"ContainerDied","Data":"d6a1091daa4799797dee9b58442e0ba0b7323eb2ad89f0846d87a12e08c24ab7"} Jul 29 18:42:45.803712 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.803561 2574 scope.go:117] "RemoveContainer" containerID="2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f" Jul 29 18:42:45.815166 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.815144 2574 scope.go:117] "RemoveContainer" containerID="2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f" Jul 29 18:42:45.815429 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:42:45.815412 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f\": container with ID starting with 2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f not found: ID does not exist" containerID="2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f" Jul 29 18:42:45.815493 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.815438 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f"} err="failed to get container status \"2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f\": rpc error: code = NotFound desc = could not find container \"2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f\": container with ID starting with 2701eb8d0031e19d7487d50d83f4a455bb4aed6887bc93cb8dc0c84ab89f3f2f not found: ID does not exist" Jul 29 18:42:45.831112 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.831066 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk"] Jul 29 18:42:45.836021 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:45.835997 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-q9zvk"] Jul 29 18:42:46.766447 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:42:46.766402 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8f0431-6ba4-4f37-bb78-675617e72fdc" path="/var/lib/kubelet/pods/6e8f0431-6ba4-4f37-bb78-675617e72fdc/volumes" Jul 29 18:43:01.339268 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.339237 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw"] Jul 29 18:43:01.339666 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.339644 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8f0431-6ba4-4f37-bb78-675617e72fdc" containerName="manager" Jul 29 18:43:01.339666 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.339662 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8f0431-6ba4-4f37-bb78-675617e72fdc" containerName="manager" Jul 29 18:43:01.339774 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.339761 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8f0431-6ba4-4f37-bb78-675617e72fdc" containerName="manager" Jul 29 18:43:01.344443 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.344422 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.346974 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.346955 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jtn4l\"" Jul 29 18:43:01.354656 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.354635 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw"] Jul 29 18:43:01.398677 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398648 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.398828 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.398828 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398711 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.398828 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.398828 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.398828 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398780 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.399038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398859 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dftq\" (UniqueName: \"kubernetes.io/projected/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-kube-api-access-7dftq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.399038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.399038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.398925 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500002 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dftq\" (UniqueName: \"kubernetes.io/projected/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-kube-api-access-7dftq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500040 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500077 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500225 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500265 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500259 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500287 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500461 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500521 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500560 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500535 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.500963 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.500910 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.501061 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.501012 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.502792 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.502769 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.502991 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.502973 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.508257 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.508226 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dftq\" (UniqueName: \"kubernetes.io/projected/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-kube-api-access-7dftq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.508377 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.508359 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6dcf6e36-4709-4f76-8190-0ed9dfdd4db8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-wlxnw\" (UID: \"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.655468 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.655391 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:01.793505 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.793476 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw"] Jul 29 18:43:01.796053 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:43:01.796025 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcf6e36_4709_4f76_8190_0ed9dfdd4db8.slice/crio-d5377fd9efdf4bcb114a7bf3ea7383501049cef73a1e25a1d5f22f727d7badff WatchSource:0}: Error finding container d5377fd9efdf4bcb114a7bf3ea7383501049cef73a1e25a1d5f22f727d7badff: Status 404 returned error can't find the container with id d5377fd9efdf4bcb114a7bf3ea7383501049cef73a1e25a1d5f22f727d7badff Jul 29 18:43:01.798071 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.798041 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:43:01.798163 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.798115 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:43:01.798228 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.798162 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:43:01.873716 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.873674 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" event={"ID":"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8","Type":"ContainerStarted","Data":"1069ee372aeb4641aa14336a95dfcca7c7bbf57fd2ffe6e015c879b3b4be4620"} Jul 29 18:43:01.873858 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.873752 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" event={"ID":"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8","Type":"ContainerStarted","Data":"d5377fd9efdf4bcb114a7bf3ea7383501049cef73a1e25a1d5f22f727d7badff"} Jul 29 18:43:01.895630 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:01.895560 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" podStartSLOduration=0.895537114 podStartE2EDuration="895.537114ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:01.894041054 +0000 UTC m=+521.606952546" watchObservedRunningTime="2026-07-29 18:43:01.895537114 +0000 UTC m=+521.608448606" Jul 29 18:43:02.656330 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:02.656292 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:02.661047 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:02.661024 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:02.877986 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:02.877956 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:02.879068 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:02.879053 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:43:03.945761 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:03.945724 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-jxzjk"] Jul 29 18:43:03.948249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:03.948220 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:03.950953 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:03.950905 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7x5zw\"" Jul 29 18:43:03.954181 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:03.954160 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jxzjk"] Jul 29 18:43:04.022802 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:04.022769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tk6r\" (UniqueName: \"kubernetes.io/projected/9ae5806a-f963-44d6-98c9-7bd995ba6d9b-kube-api-access-6tk6r\") pod \"authorino-75d556bd77-jxzjk\" (UID: \"9ae5806a-f963-44d6-98c9-7bd995ba6d9b\") " pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:04.123660 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:04.123631 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tk6r\" (UniqueName: \"kubernetes.io/projected/9ae5806a-f963-44d6-98c9-7bd995ba6d9b-kube-api-access-6tk6r\") pod \"authorino-75d556bd77-jxzjk\" (UID: \"9ae5806a-f963-44d6-98c9-7bd995ba6d9b\") " pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:04.133167 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:04.133137 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tk6r\" (UniqueName: \"kubernetes.io/projected/9ae5806a-f963-44d6-98c9-7bd995ba6d9b-kube-api-access-6tk6r\") pod \"authorino-75d556bd77-jxzjk\" (UID: \"9ae5806a-f963-44d6-98c9-7bd995ba6d9b\") " pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:04.259120 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:04.259030 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:04.388804 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:04.388774 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jxzjk"] Jul 29 18:43:04.392007 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:43:04.391974 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9ae5806a_f963_44d6_98c9_7bd995ba6d9b.slice/crio-47aed081ea5a392014c8b1df59ee875171f5f335a13713749bf0e63a5806b8e0 WatchSource:0}: Error finding container 47aed081ea5a392014c8b1df59ee875171f5f335a13713749bf0e63a5806b8e0: Status 404 returned error can't find the container with id 47aed081ea5a392014c8b1df59ee875171f5f335a13713749bf0e63a5806b8e0 Jul 29 18:43:04.889306 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:04.889271 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jxzjk" event={"ID":"9ae5806a-f963-44d6-98c9-7bd995ba6d9b","Type":"ContainerStarted","Data":"47aed081ea5a392014c8b1df59ee875171f5f335a13713749bf0e63a5806b8e0"} Jul 29 18:43:07.907237 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:07.907143 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jxzjk" event={"ID":"9ae5806a-f963-44d6-98c9-7bd995ba6d9b","Type":"ContainerStarted","Data":"5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef"} Jul 29 18:43:07.923140 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:07.923091 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-jxzjk" podStartSLOduration=1.89250118 podStartE2EDuration="4.923075445s" totalImagesPullingTime="3.030574265s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:07.921150127 +0000 UTC m=+527.634061618" watchObservedRunningTime="2026-07-29 18:43:07.923075445 +0000 UTC m=+527.635986934" Jul 29 18:43:34.063571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:34.063510 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jxzjk"] Jul 29 18:43:34.064116 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:34.063765 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-jxzjk" podUID="9ae5806a-f963-44d6-98c9-7bd995ba6d9b" containerName="authorino" containerID="cri-o://5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef" gracePeriod=30 Jul 29 18:43:34.336768 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:34.336738 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:34.381542 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:34.381512 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tk6r\" (UniqueName: \"kubernetes.io/projected/9ae5806a-f963-44d6-98c9-7bd995ba6d9b-kube-api-access-6tk6r\") pod \"9ae5806a-f963-44d6-98c9-7bd995ba6d9b\" (UID: \"9ae5806a-f963-44d6-98c9-7bd995ba6d9b\") " Jul 29 18:43:34.383770 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:34.383747 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ae5806a-f963-44d6-98c9-7bd995ba6d9b-kube-api-access-6tk6r" (OuterVolumeSpecName: "kube-api-access-6tk6r") pod "9ae5806a-f963-44d6-98c9-7bd995ba6d9b" (UID: "9ae5806a-f963-44d6-98c9-7bd995ba6d9b"). InnerVolumeSpecName "kube-api-access-6tk6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:43:34.482183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:34.482144 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tk6r\" (UniqueName: \"kubernetes.io/projected/9ae5806a-f963-44d6-98c9-7bd995ba6d9b-kube-api-access-6tk6r\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:43:35.034960 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.034853 2574 generic.go:356] "Generic (PLEG): container finished" podID="9ae5806a-f963-44d6-98c9-7bd995ba6d9b" containerID="5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef" exitCode=0 Jul 29 18:43:35.034960 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.034895 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-jxzjk" Jul 29 18:43:35.035144 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.034953 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jxzjk" event={"ID":"9ae5806a-f963-44d6-98c9-7bd995ba6d9b","Type":"ContainerDied","Data":"5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef"} Jul 29 18:43:35.035144 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.034999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-jxzjk" event={"ID":"9ae5806a-f963-44d6-98c9-7bd995ba6d9b","Type":"ContainerDied","Data":"47aed081ea5a392014c8b1df59ee875171f5f335a13713749bf0e63a5806b8e0"} Jul 29 18:43:35.035144 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.035020 2574 scope.go:117] "RemoveContainer" containerID="5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef" Jul 29 18:43:35.043206 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.043190 2574 scope.go:117] "RemoveContainer" containerID="5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef" Jul 29 18:43:35.043474 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:43:35.043455 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef\": container with ID starting with 5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef not found: ID does not exist" containerID="5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef" Jul 29 18:43:35.043518 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.043482 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef"} err="failed to get container status \"5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef\": rpc error: code = NotFound desc = could not find container \"5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef\": container with ID starting with 5de25606de6eeeaa74880b643a568dffc5f46766cb9ee50df21db08e25846cef not found: ID does not exist" Jul 29 18:43:35.050651 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.050619 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jxzjk"] Jul 29 18:43:35.054635 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:35.054609 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-jxzjk"] Jul 29 18:43:36.767244 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:36.767206 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ae5806a-f963-44d6-98c9-7bd995ba6d9b" path="/var/lib/kubelet/pods/9ae5806a-f963-44d6-98c9-7bd995ba6d9b/volumes" Jul 29 18:43:39.118461 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.118427 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-795d4b9d8f-7ndb6"] Jul 29 18:43:39.118820 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.118763 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ae5806a-f963-44d6-98c9-7bd995ba6d9b" containerName="authorino" Jul 29 18:43:39.118820 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.118775 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ae5806a-f963-44d6-98c9-7bd995ba6d9b" containerName="authorino" Jul 29 18:43:39.118894 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.118830 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ae5806a-f963-44d6-98c9-7bd995ba6d9b" containerName="authorino" Jul 29 18:43:39.121386 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.121370 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.124111 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.124089 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 18:43:39.124241 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.124187 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 18:43:39.124324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.124272 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2vcxp\"" Jul 29 18:43:39.134623 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.134597 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-795d4b9d8f-7ndb6"] Jul 29 18:43:39.223002 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.222967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbrjz\" (UniqueName: \"kubernetes.io/projected/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-kube-api-access-xbrjz\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.223183 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.223055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.321289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.321252 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d7f7957b6-smrxf"] Jul 29 18:43:39.323547 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.323530 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.323792 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.323774 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbrjz\" (UniqueName: \"kubernetes.io/projected/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-kube-api-access-xbrjz\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.323834 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.323825 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.323954 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:43:39.323931 2574 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 29 18:43:39.324088 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:43:39.323984 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert podName:e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153 nodeName:}" failed. No retries permitted until 2026-07-29 18:43:39.823969781 +0000 UTC m=+559.536881249 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert") pod "maas-controller-795d4b9d8f-7ndb6" (UID: "e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153") : secret "maas-controller-webhook-cert" not found Jul 29 18:43:39.334108 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.334083 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-smrxf"] Jul 29 18:43:39.336098 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.336072 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbrjz\" (UniqueName: \"kubernetes.io/projected/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-kube-api-access-xbrjz\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.424789 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.424738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c304cb-1d52-4d2e-9661-a710427c5d24-webhook-cert\") pod \"maas-controller-d7f7957b6-smrxf\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.425105 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.424865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2bdd\" (UniqueName: \"kubernetes.io/projected/f6c304cb-1d52-4d2e-9661-a710427c5d24-kube-api-access-l2bdd\") pod \"maas-controller-d7f7957b6-smrxf\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.526248 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.526217 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c304cb-1d52-4d2e-9661-a710427c5d24-webhook-cert\") pod \"maas-controller-d7f7957b6-smrxf\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.526410 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.526273 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2bdd\" (UniqueName: \"kubernetes.io/projected/f6c304cb-1d52-4d2e-9661-a710427c5d24-kube-api-access-l2bdd\") pod \"maas-controller-d7f7957b6-smrxf\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.528790 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.528762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c304cb-1d52-4d2e-9661-a710427c5d24-webhook-cert\") pod \"maas-controller-d7f7957b6-smrxf\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.535626 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.535605 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2bdd\" (UniqueName: \"kubernetes.io/projected/f6c304cb-1d52-4d2e-9661-a710427c5d24-kube-api-access-l2bdd\") pod \"maas-controller-d7f7957b6-smrxf\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.633977 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.633948 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:39.762008 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.761945 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-smrxf"] Jul 29 18:43:39.764195 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:43:39.764164 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6c304cb_1d52_4d2e_9661_a710427c5d24.slice/crio-005c82b2c09cbee10c2e38b30621be356a9161b0348084aa9584be65ed112567 WatchSource:0}: Error finding container 005c82b2c09cbee10c2e38b30621be356a9161b0348084aa9584be65ed112567: Status 404 returned error can't find the container with id 005c82b2c09cbee10c2e38b30621be356a9161b0348084aa9584be65ed112567 Jul 29 18:43:39.829167 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.829136 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:39.831493 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:39.831468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert\") pod \"maas-controller-795d4b9d8f-7ndb6\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:40.031385 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:40.031296 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:40.058063 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:40.058030 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-smrxf" event={"ID":"f6c304cb-1d52-4d2e-9661-a710427c5d24","Type":"ContainerStarted","Data":"005c82b2c09cbee10c2e38b30621be356a9161b0348084aa9584be65ed112567"} Jul 29 18:43:40.163512 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:40.163490 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-795d4b9d8f-7ndb6"] Jul 29 18:43:40.165301 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:43:40.165278 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4e7f57b_5ea6_4e1d_af4c_8dd8ef5e2153.slice/crio-903e5226adb95fffb22b0e680be1ffd8327625bbf76658437fea8837c9ce31cf WatchSource:0}: Error finding container 903e5226adb95fffb22b0e680be1ffd8327625bbf76658437fea8837c9ce31cf: Status 404 returned error can't find the container with id 903e5226adb95fffb22b0e680be1ffd8327625bbf76658437fea8837c9ce31cf Jul 29 18:43:41.063853 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:41.063820 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" event={"ID":"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153","Type":"ContainerStarted","Data":"903e5226adb95fffb22b0e680be1ffd8327625bbf76658437fea8837c9ce31cf"} Jul 29 18:43:43.073669 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:43.073630 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" event={"ID":"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153","Type":"ContainerStarted","Data":"bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101"} Jul 29 18:43:43.074179 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:43.073742 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:43.077767 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:43.077736 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-smrxf" event={"ID":"f6c304cb-1d52-4d2e-9661-a710427c5d24","Type":"ContainerStarted","Data":"20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e"} Jul 29 18:43:43.078029 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:43.077973 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:43.095603 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:43.095556 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" podStartSLOduration=2.138758656 podStartE2EDuration="4.095542989s" totalImagesPullingTime="1.956784333s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:43.093909568 +0000 UTC m=+562.806821051" watchObservedRunningTime="2026-07-29 18:43:43.095542989 +0000 UTC m=+562.808454479" Jul 29 18:43:43.113219 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:43.113171 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d7f7957b6-smrxf" podStartSLOduration=1.75584575 podStartE2EDuration="4.113155947s" totalImagesPullingTime="2.357310197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:43:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:43:43.112169309 +0000 UTC m=+562.825080802" watchObservedRunningTime="2026-07-29 18:43:43.113155947 +0000 UTC m=+562.826067438" Jul 29 18:43:54.085999 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.085957 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:54.086556 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.086048 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:43:54.154249 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.154203 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-795d4b9d8f-7ndb6"] Jul 29 18:43:54.154491 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.154466 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" podUID="e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" containerName="manager" containerID="cri-o://bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101" gracePeriod=30 Jul 29 18:43:54.401247 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.401222 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:54.458230 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.458192 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbrjz\" (UniqueName: \"kubernetes.io/projected/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-kube-api-access-xbrjz\") pod \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " Jul 29 18:43:54.458230 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.458234 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert\") pod \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\" (UID: \"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153\") " Jul 29 18:43:54.460613 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.460577 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-kube-api-access-xbrjz" (OuterVolumeSpecName: "kube-api-access-xbrjz") pod "e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" (UID: "e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153"). InnerVolumeSpecName "kube-api-access-xbrjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:43:54.460613 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.460582 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" (UID: "e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:43:54.559628 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.559590 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbrjz\" (UniqueName: \"kubernetes.io/projected/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-kube-api-access-xbrjz\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:43:54.559628 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:54.559620 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153-webhook-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:43:55.132145 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.132067 2574 generic.go:356] "Generic (PLEG): container finished" podID="e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" containerID="bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101" exitCode=0 Jul 29 18:43:55.132145 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.132107 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" Jul 29 18:43:55.132571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.132145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" event={"ID":"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153","Type":"ContainerDied","Data":"bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101"} Jul 29 18:43:55.132571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.132197 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-795d4b9d8f-7ndb6" event={"ID":"e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153","Type":"ContainerDied","Data":"903e5226adb95fffb22b0e680be1ffd8327625bbf76658437fea8837c9ce31cf"} Jul 29 18:43:55.132571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.132223 2574 scope.go:117] "RemoveContainer" containerID="bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101" Jul 29 18:43:55.140412 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.140396 2574 scope.go:117] "RemoveContainer" containerID="bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101" Jul 29 18:43:55.140690 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:43:55.140670 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101\": container with ID starting with bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101 not found: ID does not exist" containerID="bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101" Jul 29 18:43:55.140773 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.140701 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101"} err="failed to get container status \"bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101\": rpc error: code = NotFound desc = could not find container \"bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101\": container with ID starting with bf49e75a35cb58a3a14753b716d463eef8627d4bbd536bb87c2933d773fa0101 not found: ID does not exist" Jul 29 18:43:55.148284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.148263 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-795d4b9d8f-7ndb6"] Jul 29 18:43:55.150757 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:55.150737 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-795d4b9d8f-7ndb6"] Jul 29 18:43:56.766510 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:43:56.766479 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" path="/var/lib/kubelet/pods/e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153/volumes" Jul 29 18:44:14.858910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.858876 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb"] Jul 29 18:44:14.859414 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.859239 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" containerName="manager" Jul 29 18:44:14.859414 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.859251 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" containerName="manager" Jul 29 18:44:14.859414 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.859331 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4e7f57b-5ea6-4e1d-af4c-8dd8ef5e2153" containerName="manager" Jul 29 18:44:14.862285 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.862266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:14.866117 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.866092 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mglnh\"" Jul 29 18:44:14.866117 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.866112 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 29 18:44:14.866299 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.866120 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Jul 29 18:44:14.866299 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.866140 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 29 18:44:14.866299 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.866112 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Jul 29 18:44:14.873498 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.873477 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb"] Jul 29 18:44:14.938610 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.938571 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41337600-2a96-48de-ae81-bce4a122a3ff-openshift-service-ca\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:14.938803 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.938630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41337600-2a96-48de-ae81-bce4a122a3ff-maas-api-tls\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:14.938803 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.938719 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41337600-2a96-48de-ae81-bce4a122a3ff-metrics-certs\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:14.938803 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:14.938769 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zspz\" (UniqueName: \"kubernetes.io/projected/41337600-2a96-48de-ae81-bce4a122a3ff-kube-api-access-6zspz\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.039901 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.039858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41337600-2a96-48de-ae81-bce4a122a3ff-metrics-certs\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.040128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.039910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zspz\" (UniqueName: \"kubernetes.io/projected/41337600-2a96-48de-ae81-bce4a122a3ff-kube-api-access-6zspz\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.040128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.039983 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41337600-2a96-48de-ae81-bce4a122a3ff-openshift-service-ca\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.040128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.040061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41337600-2a96-48de-ae81-bce4a122a3ff-maas-api-tls\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.040866 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.040799 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/41337600-2a96-48de-ae81-bce4a122a3ff-openshift-service-ca\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.042852 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.042776 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/41337600-2a96-48de-ae81-bce4a122a3ff-metrics-certs\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.043508 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.043480 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/41337600-2a96-48de-ae81-bce4a122a3ff-maas-api-tls\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.060994 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.060948 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zspz\" (UniqueName: \"kubernetes.io/projected/41337600-2a96-48de-ae81-bce4a122a3ff-kube-api-access-6zspz\") pod \"maas-api-6985f9bc9c-hgdqb\" (UID: \"41337600-2a96-48de-ae81-bce4a122a3ff\") " pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.173324 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.173289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:15.322845 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.322818 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb"] Jul 29 18:44:15.324950 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:44:15.324905 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41337600_2a96_48de_ae81_bce4a122a3ff.slice/crio-14a86d4a47974da2125638f48a1e4dbec7fcb3de80314bdae334e922b510bc21 WatchSource:0}: Error finding container 14a86d4a47974da2125638f48a1e4dbec7fcb3de80314bdae334e922b510bc21: Status 404 returned error can't find the container with id 14a86d4a47974da2125638f48a1e4dbec7fcb3de80314bdae334e922b510bc21 Jul 29 18:44:15.368857 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.368826 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd"] Jul 29 18:44:15.379497 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.379464 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-bps2k"] Jul 29 18:44:15.379671 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.379630 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.383650 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.382692 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-5pgxq\"" Jul 29 18:44:15.383650 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.382782 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 29 18:44:15.387867 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.387833 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd"] Jul 29 18:44:15.388003 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.387877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-bps2k"] Jul 29 18:44:15.388003 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.387991 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.443334 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.443247 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2eeae5-b163-4f47-9581-096c74aaafca-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-bps2k\" (UID: \"8b2eeae5-b163-4f47-9581-096c74aaafca\") " pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.443519 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.443493 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-855pr\" (UniqueName: \"kubernetes.io/projected/8b2eeae5-b163-4f47-9581-096c74aaafca-kube-api-access-855pr\") pod \"payload-processing-5cc6cd444b-bps2k\" (UID: \"8b2eeae5-b163-4f47-9581-096c74aaafca\") " pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.544367 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.544329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2eeae5-b163-4f47-9581-096c74aaafca-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-bps2k\" (UID: \"8b2eeae5-b163-4f47-9581-096c74aaafca\") " pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.544558 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.544390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbplg\" (UniqueName: \"kubernetes.io/projected/f96d6bae-87b7-4c81-9ca1-fe2cdd75db18-kube-api-access-mbplg\") pod \"payload-pre-processing-687dbb6cf6-jqmvd\" (UID: \"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.544558 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.544419 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96d6bae-87b7-4c81-9ca1-fe2cdd75db18-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jqmvd\" (UID: \"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.544558 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.544445 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-855pr\" (UniqueName: \"kubernetes.io/projected/8b2eeae5-b163-4f47-9581-096c74aaafca-kube-api-access-855pr\") pod \"payload-processing-5cc6cd444b-bps2k\" (UID: \"8b2eeae5-b163-4f47-9581-096c74aaafca\") " pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.545083 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.545063 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b2eeae5-b163-4f47-9581-096c74aaafca-plugins-config-volume\") pod \"payload-processing-5cc6cd444b-bps2k\" (UID: \"8b2eeae5-b163-4f47-9581-096c74aaafca\") " pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.552817 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.552793 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-855pr\" (UniqueName: \"kubernetes.io/projected/8b2eeae5-b163-4f47-9581-096c74aaafca-kube-api-access-855pr\") pod \"payload-processing-5cc6cd444b-bps2k\" (UID: \"8b2eeae5-b163-4f47-9581-096c74aaafca\") " pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.645230 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.645192 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbplg\" (UniqueName: \"kubernetes.io/projected/f96d6bae-87b7-4c81-9ca1-fe2cdd75db18-kube-api-access-mbplg\") pod \"payload-pre-processing-687dbb6cf6-jqmvd\" (UID: \"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.645419 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.645252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96d6bae-87b7-4c81-9ca1-fe2cdd75db18-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jqmvd\" (UID: \"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.646001 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.645981 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f96d6bae-87b7-4c81-9ca1-fe2cdd75db18-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-jqmvd\" (UID: \"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.653847 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.653816 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbplg\" (UniqueName: \"kubernetes.io/projected/f96d6bae-87b7-4c81-9ca1-fe2cdd75db18-kube-api-access-mbplg\") pod \"payload-pre-processing-687dbb6cf6-jqmvd\" (UID: \"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.696558 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.696469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:15.704142 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.704115 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:15.862478 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.862422 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd"] Jul 29 18:44:15.865324 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:44:15.865297 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf96d6bae_87b7_4c81_9ca1_fe2cdd75db18.slice/crio-28787c52600df33ffef03ea70c4cb5cf40b49f283cc0181fa7abe44829bbb5c9 WatchSource:0}: Error finding container 28787c52600df33ffef03ea70c4cb5cf40b49f283cc0181fa7abe44829bbb5c9: Status 404 returned error can't find the container with id 28787c52600df33ffef03ea70c4cb5cf40b49f283cc0181fa7abe44829bbb5c9 Jul 29 18:44:15.885836 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:15.885799 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5cc6cd444b-bps2k"] Jul 29 18:44:15.890286 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:44:15.890251 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b2eeae5_b163_4f47_9581_096c74aaafca.slice/crio-81167261d366d33c45564e6670724010ac69c5c00a3b70132d0970fec9b8cd78 WatchSource:0}: Error finding container 81167261d366d33c45564e6670724010ac69c5c00a3b70132d0970fec9b8cd78: Status 404 returned error can't find the container with id 81167261d366d33c45564e6670724010ac69c5c00a3b70132d0970fec9b8cd78 Jul 29 18:44:16.228329 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:16.228288 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" event={"ID":"8b2eeae5-b163-4f47-9581-096c74aaafca","Type":"ContainerStarted","Data":"81167261d366d33c45564e6670724010ac69c5c00a3b70132d0970fec9b8cd78"} Jul 29 18:44:16.229645 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:16.229608 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" event={"ID":"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18","Type":"ContainerStarted","Data":"28787c52600df33ffef03ea70c4cb5cf40b49f283cc0181fa7abe44829bbb5c9"} Jul 29 18:44:16.230834 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:16.230811 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" event={"ID":"41337600-2a96-48de-ae81-bce4a122a3ff","Type":"ContainerStarted","Data":"14a86d4a47974da2125638f48a1e4dbec7fcb3de80314bdae334e922b510bc21"} Jul 29 18:44:20.256797 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.256758 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" event={"ID":"41337600-2a96-48de-ae81-bce4a122a3ff","Type":"ContainerStarted","Data":"4fa14802e413167beee39f378526e3ca4213c1f843b2045b3320937d9a78db37"} Jul 29 18:44:20.257243 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.256959 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:20.258436 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.258408 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" event={"ID":"8b2eeae5-b163-4f47-9581-096c74aaafca","Type":"ContainerStarted","Data":"d3f611522948b2cf1ed21dbfaa20d6797788d1a87aa7f84892cad9d427da8742"} Jul 29 18:44:20.258596 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.258516 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:20.259945 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.259906 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" event={"ID":"f96d6bae-87b7-4c81-9ca1-fe2cdd75db18","Type":"ContainerStarted","Data":"419954a5f2171b4bbf2d1d9b088da23ccf34b85c099baeb856b4ee695920b823"} Jul 29 18:44:20.260066 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.260016 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:20.274764 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.274717 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" podStartSLOduration=2.2897358580000002 podStartE2EDuration="6.274701813s" totalImagesPullingTime="3.984965955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:20.273298147 +0000 UTC m=+599.986209640" watchObservedRunningTime="2026-07-29 18:44:20.274701813 +0000 UTC m=+599.987613304" Jul 29 18:44:20.290270 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.290215 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" podStartSLOduration=1.81585437 podStartE2EDuration="5.290194401s" totalImagesPullingTime="3.474340031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:20.287815327 +0000 UTC m=+600.000726816" watchObservedRunningTime="2026-07-29 18:44:20.290194401 +0000 UTC m=+600.003105892" Jul 29 18:44:20.307454 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:20.307397 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" podStartSLOduration=1.863011647 podStartE2EDuration="5.307376914s" totalImagesPullingTime="3.444365267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:20.305310419 +0000 UTC m=+600.018221910" watchObservedRunningTime="2026-07-29 18:44:20.307376914 +0000 UTC m=+600.020288405" Jul 29 18:44:23.276244 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:23.276169 2574 generic.go:356] "Generic (PLEG): container finished" podID="6dcf6e36-4709-4f76-8190-0ed9dfdd4db8" containerID="1069ee372aeb4641aa14336a95dfcca7c7bbf57fd2ffe6e015c879b3b4be4620" exitCode=0 Jul 29 18:44:23.276244 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:23.276208 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" event={"ID":"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8","Type":"ContainerDied","Data":"1069ee372aeb4641aa14336a95dfcca7c7bbf57fd2ffe6e015c879b3b4be4620"} Jul 29 18:44:23.276656 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:23.276564 2574 scope.go:117] "RemoveContainer" containerID="1069ee372aeb4641aa14336a95dfcca7c7bbf57fd2ffe6e015c879b3b4be4620" Jul 29 18:44:23.277501 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:23.277472 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:44:23.277553 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:23.277539 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:44:23.277600 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:23.277576 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 18:44:24.284147 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:24.284110 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" event={"ID":"6dcf6e36-4709-4f76-8190-0ed9dfdd4db8","Type":"ContainerStarted","Data":"3458d6bf5319bb2dbb71e816b7c3d05ffb9c8a41812b7ebe9e1d8605eba7e9ca"} Jul 29 18:44:24.656395 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:24.656311 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:44:24.656547 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:24.656522 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wlxnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 29 18:44:24.656606 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:24.656587 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" podUID="6dcf6e36-4709-4f76-8190-0ed9dfdd4db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 29 18:44:25.655938 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:25.655888 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wlxnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 29 18:44:25.656356 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:25.655977 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" podUID="6dcf6e36-4709-4f76-8190-0ed9dfdd4db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 29 18:44:26.270619 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:26.270588 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6985f9bc9c-hgdqb" Jul 29 18:44:26.656303 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:26.656215 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wlxnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 29 18:44:26.656303 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:26.656274 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" podUID="6dcf6e36-4709-4f76-8190-0ed9dfdd4db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 29 18:44:27.655830 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:27.655794 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wlxnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 29 18:44:27.656023 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:27.655857 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" podUID="6dcf6e36-4709-4f76-8190-0ed9dfdd4db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 29 18:44:28.655848 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:28.655812 2574 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-wlxnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" start-of-body= Jul 29 18:44:28.656268 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:28.655868 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" podUID="6dcf6e36-4709-4f76-8190-0ed9dfdd4db8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.29:15021/healthz/ready\": dial tcp 10.133.0.29:15021: connect: connection refused" Jul 29 18:44:29.660571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:29.660542 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:44:29.661071 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:29.660818 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:44:29.661478 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:29.661459 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-wlxnw" Jul 29 18:44:31.267100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:31.267070 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-jqmvd" Jul 29 18:44:31.267482 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:31.267157 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5cc6cd444b-bps2k" Jul 29 18:44:52.259519 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.259387 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df"] Jul 29 18:44:52.311120 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.311079 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df"] Jul 29 18:44:52.311272 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.311213 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.315204 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.315182 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 29 18:44:52.315406 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.315182 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 29 18:44:52.315406 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.315188 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-479pt\"" Jul 29 18:44:52.315406 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.315266 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 29 18:44:52.472720 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.472684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.472887 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.472741 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.472887 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.472809 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.472887 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.472881 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zlwp\" (UniqueName: \"kubernetes.io/projected/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-kube-api-access-7zlwp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.473019 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.472906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.473019 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.472943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574344 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574268 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574344 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574315 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574517 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574354 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zlwp\" (UniqueName: \"kubernetes.io/projected/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-kube-api-access-7zlwp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574517 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574380 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574517 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574517 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574418 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574793 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574770 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574793 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574786 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.574907 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.574832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.576764 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.576736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.576967 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.576952 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.586781 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.586761 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zlwp\" (UniqueName: \"kubernetes.io/projected/57fe6ad0-6337-4ad8-a6ad-056afc8268bd-kube-api-access-7zlwp\") pod \"e2e-distinct-simulated-kserve-59975d69f9-mt7df\" (UID: \"57fe6ad0-6337-4ad8-a6ad-056afc8268bd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.621708 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.621681 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:44:52.750843 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:52.750815 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df"] Jul 29 18:44:52.752811 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:44:52.752784 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fe6ad0_6337_4ad8_a6ad_056afc8268bd.slice/crio-13c8960f80c8dbcb30e7fd5a1fe160ed6f66b6b8f8ffb69f129c736b93c10d39 WatchSource:0}: Error finding container 13c8960f80c8dbcb30e7fd5a1fe160ed6f66b6b8f8ffb69f129c736b93c10d39: Status 404 returned error can't find the container with id 13c8960f80c8dbcb30e7fd5a1fe160ed6f66b6b8f8ffb69f129c736b93c10d39 Jul 29 18:44:53.426497 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:53.426432 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" event={"ID":"57fe6ad0-6337-4ad8-a6ad-056afc8268bd","Type":"ContainerStarted","Data":"13c8960f80c8dbcb30e7fd5a1fe160ed6f66b6b8f8ffb69f129c736b93c10d39"} Jul 29 18:44:54.433058 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:54.433018 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" event={"ID":"57fe6ad0-6337-4ad8-a6ad-056afc8268bd","Type":"ContainerStarted","Data":"9934409898ad87f805599608a984e79d45a61ef0ef581b14352559631dbe8c13"} Jul 29 18:44:54.450754 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:44:54.450698 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" podStartSLOduration=1.496681686 podStartE2EDuration="2.450682611s" totalImagesPullingTime="954.000925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:44:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:44:54.450290033 +0000 UTC m=+634.163201525" watchObservedRunningTime="2026-07-29 18:44:54.450682611 +0000 UTC m=+634.163594125" Jul 29 18:45:02.622325 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:02.622290 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:45:02.622726 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:02.622384 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:45:02.635003 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:02.634980 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:45:03.482081 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:03.482056 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-mt7df" Jul 29 18:45:15.971032 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:15.970994 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9"] Jul 29 18:45:15.974581 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:15.974561 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:15.977372 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:15.977350 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 29 18:45:15.988910 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:15.988881 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9"] Jul 29 18:45:16.082495 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.082464 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.082669 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.082534 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.082669 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.082590 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f994a-3752-4192-89a3-347d5154cd20-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.082669 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.082642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.082669 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.082664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdwfn\" (UniqueName: \"kubernetes.io/projected/2b2f994a-3752-4192-89a3-347d5154cd20-kube-api-access-fdwfn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.082808 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.082701 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.183991 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.183957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184157 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184005 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f994a-3752-4192-89a3-347d5154cd20-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184157 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184157 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184092 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdwfn\" (UniqueName: \"kubernetes.io/projected/2b2f994a-3752-4192-89a3-347d5154cd20-kube-api-access-fdwfn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184157 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184125 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184372 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184168 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184491 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184468 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184571 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184518 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.184694 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.184673 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.186557 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.186536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2b2f994a-3752-4192-89a3-347d5154cd20-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.186722 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.186704 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2b2f994a-3752-4192-89a3-347d5154cd20-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.192689 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.192660 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdwfn\" (UniqueName: \"kubernetes.io/projected/2b2f994a-3752-4192-89a3-347d5154cd20-kube-api-access-fdwfn\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9\" (UID: \"2b2f994a-3752-4192-89a3-347d5154cd20\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.284346 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.284281 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:16.417545 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.417523 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9"] Jul 29 18:45:16.419254 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:45:16.419227 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b2f994a_3752_4192_89a3_347d5154cd20.slice/crio-429a32d1c38b1b14f5b2b0057d45f42c575fcf2f9483be1f6954161f6db596d7 WatchSource:0}: Error finding container 429a32d1c38b1b14f5b2b0057d45f42c575fcf2f9483be1f6954161f6db596d7: Status 404 returned error can't find the container with id 429a32d1c38b1b14f5b2b0057d45f42c575fcf2f9483be1f6954161f6db596d7 Jul 29 18:45:16.420577 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.420560 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 18:45:16.531643 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:16.531612 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" event={"ID":"2b2f994a-3752-4192-89a3-347d5154cd20","Type":"ContainerStarted","Data":"429a32d1c38b1b14f5b2b0057d45f42c575fcf2f9483be1f6954161f6db596d7"} Jul 29 18:45:17.543638 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:17.543600 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" event={"ID":"2b2f994a-3752-4192-89a3-347d5154cd20","Type":"ContainerStarted","Data":"7621651acffda97d8090c164246357f34612831562cee21a5da1d00241971ec4"} Jul 29 18:45:17.562150 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:17.562108 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" podStartSLOduration=2.377675723 podStartE2EDuration="2.562094135s" totalImagesPullingTime="184.418412ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:45:17.56168222 +0000 UTC m=+657.274593727" watchObservedRunningTime="2026-07-29 18:45:17.562094135 +0000 UTC m=+657.275005625" Jul 29 18:45:26.284452 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:26.284417 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:26.284855 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:26.284481 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:26.296981 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:26.296960 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:45:26.597012 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:45:26.596932 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9" Jul 29 18:47:14.337366 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.337323 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-smrxf"] Jul 29 18:47:14.337821 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.337552 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d7f7957b6-smrxf" podUID="f6c304cb-1d52-4d2e-9661-a710427c5d24" containerName="manager" containerID="cri-o://20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e" gracePeriod=30 Jul 29 18:47:14.593548 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.593476 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:47:14.749731 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.749700 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c304cb-1d52-4d2e-9661-a710427c5d24-webhook-cert\") pod \"f6c304cb-1d52-4d2e-9661-a710427c5d24\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " Jul 29 18:47:14.749912 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.749742 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2bdd\" (UniqueName: \"kubernetes.io/projected/f6c304cb-1d52-4d2e-9661-a710427c5d24-kube-api-access-l2bdd\") pod \"f6c304cb-1d52-4d2e-9661-a710427c5d24\" (UID: \"f6c304cb-1d52-4d2e-9661-a710427c5d24\") " Jul 29 18:47:14.752057 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.752030 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6c304cb-1d52-4d2e-9661-a710427c5d24-kube-api-access-l2bdd" (OuterVolumeSpecName: "kube-api-access-l2bdd") pod "f6c304cb-1d52-4d2e-9661-a710427c5d24" (UID: "f6c304cb-1d52-4d2e-9661-a710427c5d24"). InnerVolumeSpecName "kube-api-access-l2bdd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:47:14.752057 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.752044 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6c304cb-1d52-4d2e-9661-a710427c5d24-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f6c304cb-1d52-4d2e-9661-a710427c5d24" (UID: "f6c304cb-1d52-4d2e-9661-a710427c5d24"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:47:14.850805 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.850719 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f6c304cb-1d52-4d2e-9661-a710427c5d24-webhook-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:47:14.850805 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:14.850748 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2bdd\" (UniqueName: \"kubernetes.io/projected/f6c304cb-1d52-4d2e-9661-a710427c5d24-kube-api-access-l2bdd\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:47:15.085038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.085001 2574 generic.go:356] "Generic (PLEG): container finished" podID="f6c304cb-1d52-4d2e-9661-a710427c5d24" containerID="20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e" exitCode=0 Jul 29 18:47:15.085038 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.085041 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-smrxf" event={"ID":"f6c304cb-1d52-4d2e-9661-a710427c5d24","Type":"ContainerDied","Data":"20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e"} Jul 29 18:47:15.085289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.085060 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-smrxf" Jul 29 18:47:15.085289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.085108 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-smrxf" event={"ID":"f6c304cb-1d52-4d2e-9661-a710427c5d24","Type":"ContainerDied","Data":"005c82b2c09cbee10c2e38b30621be356a9161b0348084aa9584be65ed112567"} Jul 29 18:47:15.085289 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.085130 2574 scope.go:117] "RemoveContainer" containerID="20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e" Jul 29 18:47:15.094290 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.094260 2574 scope.go:117] "RemoveContainer" containerID="20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e" Jul 29 18:47:15.094550 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:47:15.094530 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e\": container with ID starting with 20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e not found: ID does not exist" containerID="20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e" Jul 29 18:47:15.094624 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.094559 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e"} err="failed to get container status \"20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e\": rpc error: code = NotFound desc = could not find container \"20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e\": container with ID starting with 20ac5b6b9cbbf44f834bda8146e3c4bd43399ae64fbc16ffe911af16f170482e not found: ID does not exist" Jul 29 18:47:15.102185 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.102124 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-smrxf"] Jul 29 18:47:15.106442 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:15.106352 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-smrxf"] Jul 29 18:47:16.767081 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:47:16.767044 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6c304cb-1d52-4d2e-9661-a710427c5d24" path="/var/lib/kubelet/pods/f6c304cb-1d52-4d2e-9661-a710427c5d24/volumes" Jul 29 18:48:16.013088 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.013038 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d7f7957b6-nhnxv"] Jul 29 18:48:16.013781 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.013564 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6c304cb-1d52-4d2e-9661-a710427c5d24" containerName="manager" Jul 29 18:48:16.013781 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.013592 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6c304cb-1d52-4d2e-9661-a710427c5d24" containerName="manager" Jul 29 18:48:16.013781 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.013711 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6c304cb-1d52-4d2e-9661-a710427c5d24" containerName="manager" Jul 29 18:48:16.016604 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.016589 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.020142 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.020119 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 18:48:16.020142 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.020141 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2vcxp\"" Jul 29 18:48:16.020301 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.020144 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 18:48:16.026735 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.026713 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-nhnxv"] Jul 29 18:48:16.097781 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.097746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26wp\" (UniqueName: \"kubernetes.io/projected/dec2906c-c36d-4f6b-963a-f8d9306eefdb-kube-api-access-h26wp\") pod \"maas-controller-d7f7957b6-nhnxv\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.097999 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.097787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec2906c-c36d-4f6b-963a-f8d9306eefdb-webhook-cert\") pod \"maas-controller-d7f7957b6-nhnxv\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.198496 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.198454 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26wp\" (UniqueName: \"kubernetes.io/projected/dec2906c-c36d-4f6b-963a-f8d9306eefdb-kube-api-access-h26wp\") pod \"maas-controller-d7f7957b6-nhnxv\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.198684 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.198504 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec2906c-c36d-4f6b-963a-f8d9306eefdb-webhook-cert\") pod \"maas-controller-d7f7957b6-nhnxv\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.201073 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.201050 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec2906c-c36d-4f6b-963a-f8d9306eefdb-webhook-cert\") pod \"maas-controller-d7f7957b6-nhnxv\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.206967 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.206942 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26wp\" (UniqueName: \"kubernetes.io/projected/dec2906c-c36d-4f6b-963a-f8d9306eefdb-kube-api-access-h26wp\") pod \"maas-controller-d7f7957b6-nhnxv\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.327246 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.327155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:16.461043 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:16.461011 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-nhnxv"] Jul 29 18:48:16.464123 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:48:16.464096 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddec2906c_c36d_4f6b_963a_f8d9306eefdb.slice/crio-5bacb2d0b5f7f383bdbcbfd04318070e2030ab76960a7c6c2345f5d47d2a14eb WatchSource:0}: Error finding container 5bacb2d0b5f7f383bdbcbfd04318070e2030ab76960a7c6c2345f5d47d2a14eb: Status 404 returned error can't find the container with id 5bacb2d0b5f7f383bdbcbfd04318070e2030ab76960a7c6c2345f5d47d2a14eb Jul 29 18:48:17.374336 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:17.374183 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" event={"ID":"dec2906c-c36d-4f6b-963a-f8d9306eefdb","Type":"ContainerStarted","Data":"8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6"} Jul 29 18:48:17.374748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:17.374372 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:17.374748 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:17.374396 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" event={"ID":"dec2906c-c36d-4f6b-963a-f8d9306eefdb","Type":"ContainerStarted","Data":"5bacb2d0b5f7f383bdbcbfd04318070e2030ab76960a7c6c2345f5d47d2a14eb"} Jul 29 18:48:17.393751 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:17.393645 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" podStartSLOduration=1.729716 podStartE2EDuration="2.393628561s" totalImagesPullingTime="663.912561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:48:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:48:17.392763685 +0000 UTC m=+837.105675176" watchObservedRunningTime="2026-07-29 18:48:17.393628561 +0000 UTC m=+837.106540050" Jul 29 18:48:28.382065 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:28.382027 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:33.107086 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.107052 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-nhnxv"] Jul 29 18:48:33.107627 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.107231 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" podUID="dec2906c-c36d-4f6b-963a-f8d9306eefdb" containerName="manager" containerID="cri-o://8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6" gracePeriod=30 Jul 29 18:48:33.350939 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.350902 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:33.452116 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.452082 2574 generic.go:356] "Generic (PLEG): container finished" podID="dec2906c-c36d-4f6b-963a-f8d9306eefdb" containerID="8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6" exitCode=0 Jul 29 18:48:33.452284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.452132 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" Jul 29 18:48:33.452284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.452166 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" event={"ID":"dec2906c-c36d-4f6b-963a-f8d9306eefdb","Type":"ContainerDied","Data":"8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6"} Jul 29 18:48:33.452284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.452223 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-nhnxv" event={"ID":"dec2906c-c36d-4f6b-963a-f8d9306eefdb","Type":"ContainerDied","Data":"5bacb2d0b5f7f383bdbcbfd04318070e2030ab76960a7c6c2345f5d47d2a14eb"} Jul 29 18:48:33.452284 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.452254 2574 scope.go:117] "RemoveContainer" containerID="8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6" Jul 29 18:48:33.460316 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.460303 2574 scope.go:117] "RemoveContainer" containerID="8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6" Jul 29 18:48:33.460554 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:48:33.460537 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6\": container with ID starting with 8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6 not found: ID does not exist" containerID="8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6" Jul 29 18:48:33.460605 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.460563 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6"} err="failed to get container status \"8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6\": rpc error: code = NotFound desc = could not find container \"8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6\": container with ID starting with 8e8beb88f4e9dea156f6ec0caca5811a8a127838e4e2f536e327c1809814e9a6 not found: ID does not exist" Jul 29 18:48:33.462984 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.462969 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec2906c-c36d-4f6b-963a-f8d9306eefdb-webhook-cert\") pod \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " Jul 29 18:48:33.463043 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.462999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h26wp\" (UniqueName: \"kubernetes.io/projected/dec2906c-c36d-4f6b-963a-f8d9306eefdb-kube-api-access-h26wp\") pod \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\" (UID: \"dec2906c-c36d-4f6b-963a-f8d9306eefdb\") " Jul 29 18:48:33.465040 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.465017 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dec2906c-c36d-4f6b-963a-f8d9306eefdb-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "dec2906c-c36d-4f6b-963a-f8d9306eefdb" (UID: "dec2906c-c36d-4f6b-963a-f8d9306eefdb"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 18:48:33.465128 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.465081 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dec2906c-c36d-4f6b-963a-f8d9306eefdb-kube-api-access-h26wp" (OuterVolumeSpecName: "kube-api-access-h26wp") pod "dec2906c-c36d-4f6b-963a-f8d9306eefdb" (UID: "dec2906c-c36d-4f6b-963a-f8d9306eefdb"). InnerVolumeSpecName "kube-api-access-h26wp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:48:33.564070 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.564037 2574 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/dec2906c-c36d-4f6b-963a-f8d9306eefdb-webhook-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:48:33.564070 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.564062 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h26wp\" (UniqueName: \"kubernetes.io/projected/dec2906c-c36d-4f6b-963a-f8d9306eefdb-kube-api-access-h26wp\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:48:33.774558 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.774523 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-nhnxv"] Jul 29 18:48:33.778122 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:33.778090 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-nhnxv"] Jul 29 18:48:34.650979 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.650943 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d7f7957b6-2px5x"] Jul 29 18:48:34.651339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.651314 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dec2906c-c36d-4f6b-963a-f8d9306eefdb" containerName="manager" Jul 29 18:48:34.651339 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.651326 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dec2906c-c36d-4f6b-963a-f8d9306eefdb" containerName="manager" Jul 29 18:48:34.651419 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.651390 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dec2906c-c36d-4f6b-963a-f8d9306eefdb" containerName="manager" Jul 29 18:48:34.655696 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.655679 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.658173 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.658149 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 29 18:48:34.659363 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.659340 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2vcxp\"" Jul 29 18:48:34.659626 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.659391 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 29 18:48:34.662061 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.662036 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-2px5x"] Jul 29 18:48:34.767438 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.767395 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dec2906c-c36d-4f6b-963a-f8d9306eefdb" path="/var/lib/kubelet/pods/dec2906c-c36d-4f6b-963a-f8d9306eefdb/volumes" Jul 29 18:48:34.774728 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.774704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbfdn\" (UniqueName: \"kubernetes.io/projected/81ad53d3-9475-468a-b93d-934074d78159-kube-api-access-vbfdn\") pod \"maas-controller-d7f7957b6-2px5x\" (UID: \"81ad53d3-9475-468a-b93d-934074d78159\") " pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.774838 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.774742 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81ad53d3-9475-468a-b93d-934074d78159-webhook-cert\") pod \"maas-controller-d7f7957b6-2px5x\" (UID: \"81ad53d3-9475-468a-b93d-934074d78159\") " pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.875895 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.875855 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbfdn\" (UniqueName: \"kubernetes.io/projected/81ad53d3-9475-468a-b93d-934074d78159-kube-api-access-vbfdn\") pod \"maas-controller-d7f7957b6-2px5x\" (UID: \"81ad53d3-9475-468a-b93d-934074d78159\") " pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.876062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.875910 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81ad53d3-9475-468a-b93d-934074d78159-webhook-cert\") pod \"maas-controller-d7f7957b6-2px5x\" (UID: \"81ad53d3-9475-468a-b93d-934074d78159\") " pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.878423 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.878402 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/81ad53d3-9475-468a-b93d-934074d78159-webhook-cert\") pod \"maas-controller-d7f7957b6-2px5x\" (UID: \"81ad53d3-9475-468a-b93d-934074d78159\") " pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.885799 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.885771 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbfdn\" (UniqueName: \"kubernetes.io/projected/81ad53d3-9475-468a-b93d-934074d78159-kube-api-access-vbfdn\") pod \"maas-controller-d7f7957b6-2px5x\" (UID: \"81ad53d3-9475-468a-b93d-934074d78159\") " pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:34.967487 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:34.967461 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:35.102486 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:35.102467 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d7f7957b6-2px5x"] Jul 29 18:48:35.105023 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:48:35.104991 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81ad53d3_9475_468a_b93d_934074d78159.slice/crio-1490ca35f8f73bef98424ad276b91ef81d0813d65fd1c102bf1931996a614ad3 WatchSource:0}: Error finding container 1490ca35f8f73bef98424ad276b91ef81d0813d65fd1c102bf1931996a614ad3: Status 404 returned error can't find the container with id 1490ca35f8f73bef98424ad276b91ef81d0813d65fd1c102bf1931996a614ad3 Jul 29 18:48:35.468100 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:35.468067 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-2px5x" event={"ID":"81ad53d3-9475-468a-b93d-934074d78159","Type":"ContainerStarted","Data":"1490ca35f8f73bef98424ad276b91ef81d0813d65fd1c102bf1931996a614ad3"} Jul 29 18:48:36.475042 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:36.475007 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d7f7957b6-2px5x" event={"ID":"81ad53d3-9475-468a-b93d-934074d78159","Type":"ContainerStarted","Data":"196a6ab165183e66841a57722592916e982ef9077321f26a96e973c56c8dc9b7"} Jul 29 18:48:36.475417 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:36.475158 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:48:36.494550 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:36.494501 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-d7f7957b6-2px5x" podStartSLOduration=2.016326998 podStartE2EDuration="2.494485856s" totalImagesPullingTime="478.158858ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:48:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:48:36.492525555 +0000 UTC m=+856.205437062" watchObservedRunningTime="2026-07-29 18:48:36.494485856 +0000 UTC m=+856.207397370" Jul 29 18:48:47.482170 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:48:47.482137 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d7f7957b6-2px5x" Jul 29 18:53:42.047368 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:42.047259 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7"] Jul 29 18:53:42.050067 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:42.047587 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" podUID="eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" containerName="manager" containerID="cri-o://32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828" gracePeriod=10 Jul 29 18:53:45.301508 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.301483 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:53:45.367982 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.367879 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-extensions-socket-volume\") pod \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " Jul 29 18:53:45.368122 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.368003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4462l\" (UniqueName: \"kubernetes.io/projected/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-kube-api-access-4462l\") pod \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\" (UID: \"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52\") " Jul 29 18:53:45.368259 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.368236 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" (UID: "eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 18:53:45.368357 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.368343 2574 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-extensions-socket-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:53:45.370434 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.370404 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-kube-api-access-4462l" (OuterVolumeSpecName: "kube-api-access-4462l") pod "eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" (UID: "eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52"). InnerVolumeSpecName "kube-api-access-4462l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 18:53:45.469025 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.468989 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4462l\" (UniqueName: \"kubernetes.io/projected/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52-kube-api-access-4462l\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 18:53:45.930480 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.930437 2574 generic.go:356] "Generic (PLEG): container finished" podID="eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" containerID="32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828" exitCode=0 Jul 29 18:53:45.930661 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.930505 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" Jul 29 18:53:45.930661 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.930521 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" event={"ID":"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52","Type":"ContainerDied","Data":"32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828"} Jul 29 18:53:45.930661 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.930611 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7" event={"ID":"eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52","Type":"ContainerDied","Data":"1e021c47f0f41ea0fe96fa499cbed0fa704fadb3f9b9338750da6af59a7146f2"} Jul 29 18:53:45.930661 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.930635 2574 scope.go:117] "RemoveContainer" containerID="32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828" Jul 29 18:53:45.940715 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.940695 2574 scope.go:117] "RemoveContainer" containerID="32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828" Jul 29 18:53:45.940999 ip-10-0-137-210 kubenswrapper[2574]: E0729 18:53:45.940983 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828\": container with ID starting with 32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828 not found: ID does not exist" containerID="32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828" Jul 29 18:53:45.941062 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.941009 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828"} err="failed to get container status \"32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828\": rpc error: code = NotFound desc = could not find container \"32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828\": container with ID starting with 32f3ab136b79f4b16016306c8e622a92e9c1815e70d7ceb17f01c0c03d2c0828 not found: ID does not exist" Jul 29 18:53:45.956374 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.956344 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7"] Jul 29 18:53:45.964672 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:45.964645 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-z2jx7"] Jul 29 18:53:46.767302 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:53:46.767272 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" path="/var/lib/kubelet/pods/eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52/volumes" Jul 29 18:54:43.297067 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.297027 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm"] Jul 29 18:54:43.297535 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.297403 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" containerName="manager" Jul 29 18:54:43.297535 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.297414 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" containerName="manager" Jul 29 18:54:43.297535 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.297472 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="eee50a3d-e4f6-46a3-b7c7-0236ab7cfd52" containerName="manager" Jul 29 18:54:43.301830 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.301806 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.304662 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.304639 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zn4dv\"" Jul 29 18:54:43.314838 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.314811 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm"] Jul 29 18:54:43.407365 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.407327 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzppn\" (UniqueName: \"kubernetes.io/projected/dceb4e5e-70d4-4aa1-93e1-426305c5350b-kube-api-access-bzppn\") pod \"kuadrant-operator-controller-manager-86878c769f-w8cpm\" (UID: \"dceb4e5e-70d4-4aa1-93e1-426305c5350b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.407548 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.407389 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dceb4e5e-70d4-4aa1-93e1-426305c5350b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-w8cpm\" (UID: \"dceb4e5e-70d4-4aa1-93e1-426305c5350b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.508785 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.508749 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzppn\" (UniqueName: \"kubernetes.io/projected/dceb4e5e-70d4-4aa1-93e1-426305c5350b-kube-api-access-bzppn\") pod \"kuadrant-operator-controller-manager-86878c769f-w8cpm\" (UID: \"dceb4e5e-70d4-4aa1-93e1-426305c5350b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.508988 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.508805 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dceb4e5e-70d4-4aa1-93e1-426305c5350b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-w8cpm\" (UID: \"dceb4e5e-70d4-4aa1-93e1-426305c5350b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.509194 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.509173 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dceb4e5e-70d4-4aa1-93e1-426305c5350b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-w8cpm\" (UID: \"dceb4e5e-70d4-4aa1-93e1-426305c5350b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.523325 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.523297 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzppn\" (UniqueName: \"kubernetes.io/projected/dceb4e5e-70d4-4aa1-93e1-426305c5350b-kube-api-access-bzppn\") pod \"kuadrant-operator-controller-manager-86878c769f-w8cpm\" (UID: \"dceb4e5e-70d4-4aa1-93e1-426305c5350b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.612445 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.612346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:43.763630 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:43.763604 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm"] Jul 29 18:54:43.766613 ip-10-0-137-210 kubenswrapper[2574]: W0729 18:54:43.766577 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddceb4e5e_70d4_4aa1_93e1_426305c5350b.slice/crio-e669e4252b34627fa11cd565a20da82b5b835762ec8861321d0b8441f6a1d61f WatchSource:0}: Error finding container e669e4252b34627fa11cd565a20da82b5b835762ec8861321d0b8441f6a1d61f: Status 404 returned error can't find the container with id e669e4252b34627fa11cd565a20da82b5b835762ec8861321d0b8441f6a1d61f Jul 29 18:54:44.191215 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:44.191176 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" event={"ID":"dceb4e5e-70d4-4aa1-93e1-426305c5350b","Type":"ContainerStarted","Data":"7d82eafad8e0ace2b6cc0e5ddeb59c118f7255f700ed7b71b10ea42ea828403a"} Jul 29 18:54:44.191391 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:44.191230 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" event={"ID":"dceb4e5e-70d4-4aa1-93e1-426305c5350b","Type":"ContainerStarted","Data":"e669e4252b34627fa11cd565a20da82b5b835762ec8861321d0b8441f6a1d61f"} Jul 29 18:54:44.191391 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:44.191356 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 18:54:44.224666 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:44.224608 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" podStartSLOduration=1.224585348 podStartE2EDuration="1.224585348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 18:54:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 18:54:44.223233477 +0000 UTC m=+1223.936144970" watchObservedRunningTime="2026-07-29 18:54:44.224585348 +0000 UTC m=+1223.937496836" Jul 29 18:54:55.198163 ip-10-0-137-210 kubenswrapper[2574]: I0729 18:54:55.198130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-w8cpm" Jul 29 19:02:12.662860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.662822 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb"] Jul 29 19:02:12.666375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.666350 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.669405 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.669385 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e24ce08-openshift-default-dockercfg-fd5x6\"" Jul 29 19:02:12.688404 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.688374 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb"] Jul 29 19:02:12.732524 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732481 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkwp\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-kube-api-access-gbkwp\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.732524 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732526 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-envoy\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.732784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732579 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-podinfo\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.732784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732672 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-data\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.732784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732713 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-token\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.732784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732743 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-certs\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.733032 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732798 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-socket\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.733032 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732864 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istiod-ca-cert\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.733032 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.732901 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-credential-socket\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.833653 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833617 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-data\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.833855 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833662 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-token\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.833855 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833698 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-certs\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.833855 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833731 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-socket\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.833855 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istiod-ca-cert\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.833855 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-credential-socket\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834204 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833898 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkwp\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-kube-api-access-gbkwp\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834204 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.833982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-envoy\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834204 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.834027 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-podinfo\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834204 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.834091 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-data\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834204 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.834140 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-certs\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834535 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.834510 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-socket\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834656 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.834545 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-credential-socket\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.834765 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.834742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istiod-ca-cert\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.836336 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.836320 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-envoy\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.836622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.836592 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-podinfo\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.842293 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.842268 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-token\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.842475 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.842456 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkwp\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-kube-api-access-gbkwp\") pod \"e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:12.976756 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:12.976723 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:13.126313 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.126282 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb"] Jul 29 19:02:13.128975 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:02:13.128942 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41fd494d_f7c6_4473_b58a_7bbe08d1efae.slice/crio-c56b93175d26306e239ff4e7050d5802339594b61d8f606628f534f3c8fe8ef5 WatchSource:0}: Error finding container c56b93175d26306e239ff4e7050d5802339594b61d8f606628f534f3c8fe8ef5: Status 404 returned error can't find the container with id c56b93175d26306e239ff4e7050d5802339594b61d8f606628f534f3c8fe8ef5 Jul 29 19:02:13.131269 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.131233 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:13.131386 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.131302 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:13.131386 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.131332 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:13.306065 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.305960 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" event={"ID":"41fd494d-f7c6-4473-b58a-7bbe08d1efae","Type":"ContainerStarted","Data":"8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d"} Jul 29 19:02:13.306065 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.306031 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" event={"ID":"41fd494d-f7c6-4473-b58a-7bbe08d1efae","Type":"ContainerStarted","Data":"c56b93175d26306e239ff4e7050d5802339594b61d8f606628f534f3c8fe8ef5"} Jul 29 19:02:13.327858 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.327795 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podStartSLOduration=1.327778569 podStartE2EDuration="1.327778569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:13.325806733 +0000 UTC m=+1673.038718257" watchObservedRunningTime="2026-07-29 19:02:13.327778569 +0000 UTC m=+1673.040690058" Jul 29 19:02:13.977462 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:13.977419 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:14.945235 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:14.945190 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88"] Jul 29 19:02:14.954884 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:14.954847 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:14.956842 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:14.956809 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88"] Jul 29 19:02:14.960044 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:14.960015 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e24ce08-serving-cert\"" Jul 29 19:02:14.984028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:14.983996 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:15.055505 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.055468 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9897-d57e-4626-8237-35b60a361334-openshift-service-ca\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.055817 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.055788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-metrics-certs\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.056031 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.055889 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.056031 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.055937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpb8\" (UniqueName: \"kubernetes.io/projected/373a9897-d57e-4626-8237-35b60a361334-kube-api-access-wfpb8\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.156979 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.156944 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9897-d57e-4626-8237-35b60a361334-openshift-service-ca\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.157154 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.156990 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-metrics-certs\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.157154 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.157028 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.157154 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.157044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpb8\" (UniqueName: \"kubernetes.io/projected/373a9897-d57e-4626-8237-35b60a361334-kube-api-access-wfpb8\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.157154 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:15.157147 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-serving-cert: secret "maas-api-e2e-ait-0e24ce08-serving-cert" not found Jul 29 19:02:15.157377 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:15.157214 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls podName:373a9897-d57e-4626-8237-35b60a361334 nodeName:}" failed. No retries permitted until 2026-07-29 19:02:15.657196436 +0000 UTC m=+1675.370107907 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls") pod "maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" (UID: "373a9897-d57e-4626-8237-35b60a361334") : secret "maas-api-e2e-ait-0e24ce08-serving-cert" not found Jul 29 19:02:15.157601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.157580 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9897-d57e-4626-8237-35b60a361334-openshift-service-ca\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.159706 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.159685 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-metrics-certs\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.168949 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.168905 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpb8\" (UniqueName: \"kubernetes.io/projected/373a9897-d57e-4626-8237-35b60a361334-kube-api-access-wfpb8\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.314907 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.314818 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:15.315991 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.315972 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:15.662466 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.662376 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.664898 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.664869 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls\") pod \"maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.742826 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.742775 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc"] Jul 29 19:02:15.746502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.746482 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:15.749433 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.749400 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e24ce08\"" Jul 29 19:02:15.749553 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.749400 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e24ce08-dockercfg-9zd2g\"" Jul 29 19:02:15.759526 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.759494 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc"] Jul 29 19:02:15.764011 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.763985 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:15.764131 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.764114 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjxn\" (UniqueName: \"kubernetes.io/projected/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-kube-api-access-wtjxn\") pod \"payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:15.865015 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.864981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjxn\" (UniqueName: \"kubernetes.io/projected/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-kube-api-access-wtjxn\") pod \"payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:15.865178 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.865074 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:15.865772 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.865753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:15.870934 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.870897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:15.873410 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:15.873390 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjxn\" (UniqueName: \"kubernetes.io/projected/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-kube-api-access-wtjxn\") pod \"payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:16.025516 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.025489 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88"] Jul 29 19:02:16.027760 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:02:16.027732 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373a9897_d57e_4626_8237_35b60a361334.slice/crio-6028ae88010c31e4622f8bc6492e447c12814f1b0f1fdedcc37fe35e61b384a0 WatchSource:0}: Error finding container 6028ae88010c31e4622f8bc6492e447c12814f1b0f1fdedcc37fe35e61b384a0: Status 404 returned error can't find the container with id 6028ae88010c31e4622f8bc6492e447c12814f1b0f1fdedcc37fe35e61b384a0 Jul 29 19:02:16.029310 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.029293 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:02:16.064912 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.064881 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:16.207156 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.207130 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc"] Jul 29 19:02:16.210337 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:02:16.210306 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd97ad6_7730_4b5e_a4d8_9fd87bdcee20.slice/crio-1513c5294a1deba620191ae085d8990a7a27ebf139ecb73d33a02e918372f335 WatchSource:0}: Error finding container 1513c5294a1deba620191ae085d8990a7a27ebf139ecb73d33a02e918372f335: Status 404 returned error can't find the container with id 1513c5294a1deba620191ae085d8990a7a27ebf139ecb73d33a02e918372f335 Jul 29 19:02:16.321949 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.321867 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" event={"ID":"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20","Type":"ContainerStarted","Data":"7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0"} Jul 29 19:02:16.322141 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.321976 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:16.322141 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.322002 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" event={"ID":"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20","Type":"ContainerStarted","Data":"1513c5294a1deba620191ae085d8990a7a27ebf139ecb73d33a02e918372f335"} Jul 29 19:02:16.323613 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.323586 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" event={"ID":"373a9897-d57e-4626-8237-35b60a361334","Type":"ContainerStarted","Data":"6028ae88010c31e4622f8bc6492e447c12814f1b0f1fdedcc37fe35e61b384a0"} Jul 29 19:02:16.340487 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:16.340432 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" podStartSLOduration=1.34041579 podStartE2EDuration="1.34041579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:16.338564418 +0000 UTC m=+1676.051475912" watchObservedRunningTime="2026-07-29 19:02:16.34041579 +0000 UTC m=+1676.053327319" Jul 29 19:02:17.330124 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:17.330080 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" event={"ID":"373a9897-d57e-4626-8237-35b60a361334","Type":"ContainerStarted","Data":"0cef9ca28ad8aaf3bd1f278776c4aa5b52a4c4fe6db1dfc9e977b5737b8ccc61"} Jul 29 19:02:17.330613 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:17.330198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:17.350415 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:17.350368 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" podStartSLOduration=2.779006372 podStartE2EDuration="3.350354097s" totalImagesPullingTime="571.347725ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:17.348192019 +0000 UTC m=+1677.061103510" watchObservedRunningTime="2026-07-29 19:02:17.350354097 +0000 UTC m=+1677.063265587" Jul 29 19:02:23.341486 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:23.341451 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:27.332667 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:27.332636 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:30.322478 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.322444 2574 patch_prober.go:28] interesting pod/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:52714->10.133.0.41:15021: read: connection reset by peer" start-of-body= Jul 29 19:02:30.322848 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.322531 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": read tcp 10.133.0.2:52714->10.133.0.41:15021: read: connection reset by peer" Jul 29 19:02:30.397033 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.396993 2574 generic.go:356] "Generic (PLEG): container finished" podID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerID="8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d" exitCode=0 Jul 29 19:02:30.397199 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.397063 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" event={"ID":"41fd494d-f7c6-4473-b58a-7bbe08d1efae","Type":"ContainerDied","Data":"8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d"} Jul 29 19:02:30.397479 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.397466 2574 scope.go:117] "RemoveContainer" containerID="8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d" Jul 29 19:02:30.398513 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.398485 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:30.398630 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.398610 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:30.398739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:30.398664 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:31.406763 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:31.406688 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" event={"ID":"41fd494d-f7c6-4473-b58a-7bbe08d1efae","Type":"ContainerStarted","Data":"658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b"} Jul 29 19:02:31.977664 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:31.977621 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:31.977874 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:31.977850 2574 patch_prober.go:28] interesting pod/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 29 19:02:31.977973 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:31.977901 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 29 19:02:32.977316 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:32.977273 2574 patch_prober.go:28] interesting pod/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 29 19:02:32.977789 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:32.977349 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 29 19:02:33.977622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:33.977579 2574 patch_prober.go:28] interesting pod/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 29 19:02:33.978101 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:33.977647 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 29 19:02:34.977090 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:34.977057 2574 patch_prober.go:28] interesting pod/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body= Jul 29 19:02:34.977298 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:34.977128 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" Jul 29 19:02:35.981651 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:35.981616 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:35.982120 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:35.981934 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:35.982636 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:35.982619 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:45.353584 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.353506 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88"] Jul 29 19:02:45.354037 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.353719 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" podUID="373a9897-d57e-4626-8237-35b60a361334" containerName="maas-api" containerID="cri-o://0cef9ca28ad8aaf3bd1f278776c4aa5b52a4c4fe6db1dfc9e977b5737b8ccc61" gracePeriod=30 Jul 29 19:02:45.420536 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.420501 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc"] Jul 29 19:02:45.420766 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.420737 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" podUID="3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" containerName="payload-processing" containerID="cri-o://7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0" gracePeriod=30 Jul 29 19:02:45.477368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.477331 2574 generic.go:356] "Generic (PLEG): container finished" podID="373a9897-d57e-4626-8237-35b60a361334" containerID="0cef9ca28ad8aaf3bd1f278776c4aa5b52a4c4fe6db1dfc9e977b5737b8ccc61" exitCode=0 Jul 29 19:02:45.477538 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.477407 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" event={"ID":"373a9897-d57e-4626-8237-35b60a361334","Type":"ContainerDied","Data":"0cef9ca28ad8aaf3bd1f278776c4aa5b52a4c4fe6db1dfc9e977b5737b8ccc61"} Jul 29 19:02:45.699060 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.699026 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:45.726991 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.723735 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb"] Jul 29 19:02:45.726991 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.724466 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" containerID="cri-o://658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b" gracePeriod=30 Jul 29 19:02:45.726991 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.726121 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:45.871822 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.871786 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls\") pod \"373a9897-d57e-4626-8237-35b60a361334\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " Jul 29 19:02:45.872022 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.871854 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtjxn\" (UniqueName: \"kubernetes.io/projected/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-kube-api-access-wtjxn\") pod \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " Jul 29 19:02:45.872022 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.871895 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wfpb8\" (UniqueName: \"kubernetes.io/projected/373a9897-d57e-4626-8237-35b60a361334-kube-api-access-wfpb8\") pod \"373a9897-d57e-4626-8237-35b60a361334\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " Jul 29 19:02:45.872022 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.871952 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9897-d57e-4626-8237-35b60a361334-openshift-service-ca\") pod \"373a9897-d57e-4626-8237-35b60a361334\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " Jul 29 19:02:45.872022 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.872001 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-plugins-config-volume\") pod \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\" (UID: \"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20\") " Jul 29 19:02:45.872175 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.872059 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-metrics-certs\") pod \"373a9897-d57e-4626-8237-35b60a361334\" (UID: \"373a9897-d57e-4626-8237-35b60a361334\") " Jul 29 19:02:45.872444 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.872413 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373a9897-d57e-4626-8237-35b60a361334-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "373a9897-d57e-4626-8237-35b60a361334" (UID: "373a9897-d57e-4626-8237-35b60a361334"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:02:45.872612 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.872440 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" (UID: "3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:02:45.874303 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.874278 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-kube-api-access-wtjxn" (OuterVolumeSpecName: "kube-api-access-wtjxn") pod "3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" (UID: "3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20"). InnerVolumeSpecName "kube-api-access-wtjxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:02:45.874403 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.874351 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373a9897-d57e-4626-8237-35b60a361334-kube-api-access-wfpb8" (OuterVolumeSpecName: "kube-api-access-wfpb8") pod "373a9897-d57e-4626-8237-35b60a361334" (UID: "373a9897-d57e-4626-8237-35b60a361334"). InnerVolumeSpecName "kube-api-access-wfpb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:02:45.874790 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.874768 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "373a9897-d57e-4626-8237-35b60a361334" (UID: "373a9897-d57e-4626-8237-35b60a361334"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:02:45.874880 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.874785 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "373a9897-d57e-4626-8237-35b60a361334" (UID: "373a9897-d57e-4626-8237-35b60a361334"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:02:45.973324 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.973282 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/373a9897-d57e-4626-8237-35b60a361334-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:45.973324 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.973318 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-plugins-config-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:45.973324 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.973328 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:45.973324 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.973337 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/373a9897-d57e-4626-8237-35b60a361334-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:45.973601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.973345 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjxn\" (UniqueName: \"kubernetes.io/projected/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20-kube-api-access-wtjxn\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:45.973601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:45.973353 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfpb8\" (UniqueName: \"kubernetes.io/projected/373a9897-d57e-4626-8237-35b60a361334-kube-api-access-wfpb8\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:46.483297 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.483258 2574 generic.go:356] "Generic (PLEG): container finished" podID="3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" containerID="7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0" exitCode=0 Jul 29 19:02:46.483750 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.483313 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" Jul 29 19:02:46.483750 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.483339 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" event={"ID":"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20","Type":"ContainerDied","Data":"7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0"} Jul 29 19:02:46.483750 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.483424 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc" event={"ID":"3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20","Type":"ContainerDied","Data":"1513c5294a1deba620191ae085d8990a7a27ebf139ecb73d33a02e918372f335"} Jul 29 19:02:46.483750 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.483451 2574 scope.go:117] "RemoveContainer" containerID="7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0" Jul 29 19:02:46.485077 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.485016 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" Jul 29 19:02:46.485077 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.485024 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88" event={"ID":"373a9897-d57e-4626-8237-35b60a361334","Type":"ContainerDied","Data":"6028ae88010c31e4622f8bc6492e447c12814f1b0f1fdedcc37fe35e61b384a0"} Jul 29 19:02:46.493104 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.493083 2574 scope.go:117] "RemoveContainer" containerID="7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0" Jul 29 19:02:46.493386 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:46.493365 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0\": container with ID starting with 7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0 not found: ID does not exist" containerID="7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0" Jul 29 19:02:46.493466 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.493393 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0"} err="failed to get container status \"7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0\": rpc error: code = NotFound desc = could not find container \"7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0\": container with ID starting with 7140cfbfd10d71e4290dc7e823febfc807fa7c9bca39d41e5087f74a75f8b5b0 not found: ID does not exist" Jul 29 19:02:46.493466 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.493412 2574 scope.go:117] "RemoveContainer" containerID="0cef9ca28ad8aaf3bd1f278776c4aa5b52a4c4fe6db1dfc9e977b5737b8ccc61" Jul 29 19:02:46.510036 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.510010 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc"] Jul 29 19:02:46.513504 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.513482 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e24ce08-9ffd787d-77mfc"] Jul 29 19:02:46.526650 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.526621 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88"] Jul 29 19:02:46.530874 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.530852 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e24ce08-74b4b5bb7f-66g88"] Jul 29 19:02:46.767511 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.767469 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="373a9897-d57e-4626-8237-35b60a361334" path="/var/lib/kubelet/pods/373a9897-d57e-4626-8237-35b60a361334/volumes" Jul 29 19:02:46.767950 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.767905 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" path="/var/lib/kubelet/pods/3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20/volumes" Jul 29 19:02:46.902424 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902395 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w"] Jul 29 19:02:46.902765 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902752 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" containerName="payload-processing" Jul 29 19:02:46.902765 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902766 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" containerName="payload-processing" Jul 29 19:02:46.902852 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902786 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="373a9897-d57e-4626-8237-35b60a361334" containerName="maas-api" Jul 29 19:02:46.902852 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902792 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="373a9897-d57e-4626-8237-35b60a361334" containerName="maas-api" Jul 29 19:02:46.902852 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902851 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="373a9897-d57e-4626-8237-35b60a361334" containerName="maas-api" Jul 29 19:02:46.902975 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.902859 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd97ad6-7730-4b5e-a4d8-9fd87bdcee20" containerName="payload-processing" Jul 29 19:02:46.905889 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.905869 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:46.908756 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.908738 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-03561f0f-gw-openshift-default-dockercfg-z62xd\"" Jul 29 19:02:46.917511 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:46.917488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w"] Jul 29 19:02:47.084850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-certs\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.084850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5727fe90-77b5-4978-b890-6f6f9475edec-istiod-ca-cert\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.084850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084832 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-credential-socket\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.085100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084865 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-socket\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.085100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084910 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-istio-token\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.085100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084958 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-data\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.085100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.084995 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-envoy\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.085100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.085031 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7xkd\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-kube-api-access-t7xkd\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.085259 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.085099 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5727fe90-77b5-4978-b890-6f6f9475edec-istio-podinfo\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186064 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186026 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-certs\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186064 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186068 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5727fe90-77b5-4978-b890-6f6f9475edec-istiod-ca-cert\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186086 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-credential-socket\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186107 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-socket\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186229 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-istio-token\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186263 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-data\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-envoy\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186576 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7xkd\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-kube-api-access-t7xkd\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186576 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186391 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5727fe90-77b5-4978-b890-6f6f9475edec-istio-podinfo\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186576 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186457 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-certs\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186737 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186583 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-credential-socket\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186803 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186731 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-data\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186803 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186762 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-socket\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.186974 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.186944 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5727fe90-77b5-4978-b890-6f6f9475edec-istiod-ca-cert\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.188709 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.188687 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-envoy\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.188978 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.188962 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5727fe90-77b5-4978-b890-6f6f9475edec-istio-podinfo\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.204445 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.204410 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-istio-token\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.204662 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.204642 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7xkd\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-kube-api-access-t7xkd\") pod \"e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.216447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.216424 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:47.367453 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.367421 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w"] Jul 29 19:02:47.368751 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:02:47.368724 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5727fe90_77b5_4978_b890_6f6f9475edec.slice/crio-4b7a74493da77eb11cb458b3bce49b60866a3f34741e043ed9f8481570ff3584 WatchSource:0}: Error finding container 4b7a74493da77eb11cb458b3bce49b60866a3f34741e043ed9f8481570ff3584: Status 404 returned error can't find the container with id 4b7a74493da77eb11cb458b3bce49b60866a3f34741e043ed9f8481570ff3584 Jul 29 19:02:47.371004 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.370971 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:47.371078 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.371039 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:47.371078 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.371069 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:02:47.496571 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.496463 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" event={"ID":"5727fe90-77b5-4978-b890-6f6f9475edec","Type":"ContainerStarted","Data":"86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b"} Jul 29 19:02:47.496571 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.496526 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" event={"ID":"5727fe90-77b5-4978-b890-6f6f9475edec","Type":"ContainerStarted","Data":"4b7a74493da77eb11cb458b3bce49b60866a3f34741e043ed9f8481570ff3584"} Jul 29 19:02:47.523693 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:47.523622 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podStartSLOduration=1.523598295 podStartE2EDuration="1.523598295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:47.521279991 +0000 UTC m=+1707.234191483" watchObservedRunningTime="2026-07-29 19:02:47.523598295 +0000 UTC m=+1707.236509786" Jul 29 19:02:48.217458 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:48.217415 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:48.222947 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:48.222899 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:48.503231 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:48.503149 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:48.504319 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:48.504301 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:02:49.256640 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.256604 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg"] Jul 29 19:02:49.264996 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.264969 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.268548 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.268523 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-03561f0f-serving-cert\"" Jul 29 19:02:49.275025 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.274982 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg"] Jul 29 19:02:49.303460 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.303417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.303652 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.303532 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzd5g\" (UniqueName: \"kubernetes.io/projected/058afa51-f311-4379-ba49-7d334a64d6b5-kube-api-access-nzd5g\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.303652 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.303582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-metrics-certs\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.303652 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.303642 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058afa51-f311-4379-ba49-7d334a64d6b5-openshift-service-ca\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.405070 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.405037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-metrics-certs\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.405265 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.405116 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058afa51-f311-4379-ba49-7d334a64d6b5-openshift-service-ca\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.405265 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.405198 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.405265 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.405254 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzd5g\" (UniqueName: \"kubernetes.io/projected/058afa51-f311-4379-ba49-7d334a64d6b5-kube-api-access-nzd5g\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.405431 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:49.405348 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-serving-cert: secret "maas-api-e2e-migrate-03561f0f-serving-cert" not found Jul 29 19:02:49.405493 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:49.405433 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls podName:058afa51-f311-4379-ba49-7d334a64d6b5 nodeName:}" failed. No retries permitted until 2026-07-29 19:02:49.905415588 +0000 UTC m=+1709.618327057 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls") pod "maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" (UID: "058afa51-f311-4379-ba49-7d334a64d6b5") : secret "maas-api-e2e-migrate-03561f0f-serving-cert" not found Jul 29 19:02:49.405910 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.405891 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058afa51-f311-4379-ba49-7d334a64d6b5-openshift-service-ca\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.407701 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.407677 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-metrics-certs\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.415568 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.415542 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzd5g\" (UniqueName: \"kubernetes.io/projected/058afa51-f311-4379-ba49-7d334a64d6b5-kube-api-access-nzd5g\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.909406 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.909365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:49.911947 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:49.911888 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls\") pod \"maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:50.176945 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:50.176887 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:50.321161 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:50.321130 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg"] Jul 29 19:02:50.322641 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:02:50.322616 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058afa51_f311_4379_ba49_7d334a64d6b5.slice/crio-d71a7badfb98c0249ec7b4d40aac2ceea38464e02f6c09e92117a05fd54abfd9 WatchSource:0}: Error finding container d71a7badfb98c0249ec7b4d40aac2ceea38464e02f6c09e92117a05fd54abfd9: Status 404 returned error can't find the container with id d71a7badfb98c0249ec7b4d40aac2ceea38464e02f6c09e92117a05fd54abfd9 Jul 29 19:02:50.515038 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:50.514944 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" event={"ID":"058afa51-f311-4379-ba49-7d334a64d6b5","Type":"ContainerStarted","Data":"d71a7badfb98c0249ec7b4d40aac2ceea38464e02f6c09e92117a05fd54abfd9"} Jul 29 19:02:51.000751 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.000725 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:51.018813 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.018784 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-certs\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.018972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.018833 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-data\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.018972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.018875 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-token\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.018972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.018947 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-credential-socket\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.019137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.018993 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-socket\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.019137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019042 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-envoy\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.019137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019068 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-podinfo\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.019137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019096 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gbkwp\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-kube-api-access-gbkwp\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.019137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019101 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:02:51.019137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019136 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istiod-ca-cert\") pod \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\" (UID: \"41fd494d-f7c6-4473-b58a-7bbe08d1efae\") " Jul 29 19:02:51.019461 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:02:51.019461 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019324 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:02:51.019565 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019547 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.019620 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019563 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.019620 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019579 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.019891 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019633 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:02:51.019891 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.019687 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-data" (OuterVolumeSpecName: "istio-data") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:02:51.022292 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.021743 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:02:51.022292 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.022178 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:02:51.022480 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.022443 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-token" (OuterVolumeSpecName: "istio-token") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:02:51.023248 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.023220 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-kube-api-access-gbkwp" (OuterVolumeSpecName: "kube-api-access-gbkwp") pod "41fd494d-f7c6-4473-b58a-7bbe08d1efae" (UID: "41fd494d-f7c6-4473-b58a-7bbe08d1efae"). InnerVolumeSpecName "kube-api-access-gbkwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:02:51.120233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.120148 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.120233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.120176 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.120233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.120191 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbkwp\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-kube-api-access-gbkwp\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.120233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.120205 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.120233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.120218 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.120233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.120231 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41fd494d-f7c6-4473-b58a-7bbe08d1efae-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:02:51.522411 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.522372 2574 generic.go:356] "Generic (PLEG): container finished" podID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerID="658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b" exitCode=0 Jul 29 19:02:51.522592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.522423 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" Jul 29 19:02:51.522592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.522468 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" event={"ID":"41fd494d-f7c6-4473-b58a-7bbe08d1efae","Type":"ContainerDied","Data":"658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b"} Jul 29 19:02:51.522592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.522523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" event={"ID":"41fd494d-f7c6-4473-b58a-7bbe08d1efae","Type":"ContainerDied","Data":"c56b93175d26306e239ff4e7050d5802339594b61d8f606628f534f3c8fe8ef5"} Jul 29 19:02:51.522592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.522554 2574 scope.go:117] "RemoveContainer" containerID="658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b" Jul 29 19:02:51.524388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.524359 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" event={"ID":"058afa51-f311-4379-ba49-7d334a64d6b5","Type":"ContainerStarted","Data":"5b41322d6bc143fe1a36ab407649c2ad8c7be2fca41a87f1721ecd6f6256d1da"} Jul 29 19:02:51.524541 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.524433 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:02:51.533445 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.533430 2574 scope.go:117] "RemoveContainer" containerID="8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d" Jul 29 19:02:51.542391 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.542371 2574 scope.go:117] "RemoveContainer" containerID="658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b" Jul 29 19:02:51.542657 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:51.542638 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b\": container with ID starting with 658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b not found: ID does not exist" containerID="658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b" Jul 29 19:02:51.542693 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.542666 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b"} err="failed to get container status \"658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b\": rpc error: code = NotFound desc = could not find container \"658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b\": container with ID starting with 658ffb0647bdda5e21a56e14a7975064c5466d8dc993633ce8d67c19956d949b not found: ID does not exist" Jul 29 19:02:51.542693 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.542684 2574 scope.go:117] "RemoveContainer" containerID="8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d" Jul 29 19:02:51.542933 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:02:51.542897 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d\": container with ID starting with 8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d not found: ID does not exist" containerID="8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d" Jul 29 19:02:51.543032 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.542950 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d"} err="failed to get container status \"8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d\": rpc error: code = NotFound desc = could not find container \"8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d\": container with ID starting with 8f0254c23ebeff03a527282da24416ff4f140006cea6b41e8d14d47007e47a2d not found: ID does not exist" Jul 29 19:02:51.546208 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.546167 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" podStartSLOduration=1.965276705 podStartE2EDuration="2.546153226s" totalImagesPullingTime="580.876521ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:02:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:02:51.544426021 +0000 UTC m=+1711.257337522" watchObservedRunningTime="2026-07-29 19:02:51.546153226 +0000 UTC m=+1711.259064715" Jul 29 19:02:51.559572 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.559540 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb"] Jul 29 19:02:51.564062 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.564038 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb"] Jul 29 19:02:51.982781 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.982734 2574 patch_prober.go:28] interesting pod/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 19:02:51.982976 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:51.982809 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e24ce08-openshift-default-77d879779d-9ntmb" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 19:02:52.767761 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:52.767724 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" path="/var/lib/kubelet/pods/41fd494d-f7c6-4473-b58a-7bbe08d1efae/volumes" Jul 29 19:02:57.536833 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:02:57.536799 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:03:04.003585 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.002808 2574 patch_prober.go:28] interesting pod/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:35118->10.133.0.44:15021: read: connection reset by peer" start-of-body= Jul 29 19:03:04.003585 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.002890 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": read tcp 10.133.0.2:35118->10.133.0.44:15021: read: connection reset by peer" Jul 29 19:03:04.598467 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.598428 2574 generic.go:356] "Generic (PLEG): container finished" podID="5727fe90-77b5-4978-b890-6f6f9475edec" containerID="86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b" exitCode=0 Jul 29 19:03:04.598654 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.598502 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" event={"ID":"5727fe90-77b5-4978-b890-6f6f9475edec","Type":"ContainerDied","Data":"86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b"} Jul 29 19:03:04.598948 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.598910 2574 scope.go:117] "RemoveContainer" containerID="86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b" Jul 29 19:03:04.599981 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.599948 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:04.600086 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.600033 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:04.600152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:04.600082 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:05.608721 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:05.608689 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" event={"ID":"5727fe90-77b5-4978-b890-6f6f9475edec","Type":"ContainerStarted","Data":"e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3"} Jul 29 19:03:06.217492 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:06.217445 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:03:06.217710 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:06.217686 2574 patch_prober.go:28] interesting pod/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 29 19:03:06.217757 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:06.217738 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 29 19:03:07.217575 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:07.217531 2574 patch_prober.go:28] interesting pod/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 29 19:03:07.218002 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:07.217595 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 29 19:03:08.217169 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:08.217131 2574 patch_prober.go:28] interesting pod/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 29 19:03:08.217350 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:08.217202 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 29 19:03:09.217568 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:09.217526 2574 patch_prober.go:28] interesting pod/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Jul 29 19:03:09.218091 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:09.217591 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Jul 29 19:03:10.220129 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:10.220095 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:03:10.220561 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:10.220440 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:03:10.221114 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:10.221096 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:03:20.458256 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.458221 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg"] Jul 29 19:03:20.458741 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.458470 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" podUID="058afa51-f311-4379-ba49-7d334a64d6b5" containerName="maas-api" containerID="cri-o://5b41322d6bc143fe1a36ab407649c2ad8c7be2fca41a87f1721ecd6f6256d1da" gracePeriod=30 Jul 29 19:03:20.683150 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.683105 2574 generic.go:356] "Generic (PLEG): container finished" podID="058afa51-f311-4379-ba49-7d334a64d6b5" containerID="5b41322d6bc143fe1a36ab407649c2ad8c7be2fca41a87f1721ecd6f6256d1da" exitCode=0 Jul 29 19:03:20.683341 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.683201 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" event={"ID":"058afa51-f311-4379-ba49-7d334a64d6b5","Type":"ContainerDied","Data":"5b41322d6bc143fe1a36ab407649c2ad8c7be2fca41a87f1721ecd6f6256d1da"} Jul 29 19:03:20.731380 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.731354 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:03:20.796131 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.796092 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w"] Jul 29 19:03:20.796415 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.796365 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" containerID="cri-o://e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3" gracePeriod=30 Jul 29 19:03:20.821981 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.821950 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls\") pod \"058afa51-f311-4379-ba49-7d334a64d6b5\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " Jul 29 19:03:20.822146 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.822049 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-metrics-certs\") pod \"058afa51-f311-4379-ba49-7d334a64d6b5\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " Jul 29 19:03:20.822146 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.822078 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nzd5g\" (UniqueName: \"kubernetes.io/projected/058afa51-f311-4379-ba49-7d334a64d6b5-kube-api-access-nzd5g\") pod \"058afa51-f311-4379-ba49-7d334a64d6b5\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " Jul 29 19:03:20.822233 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.822159 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058afa51-f311-4379-ba49-7d334a64d6b5-openshift-service-ca\") pod \"058afa51-f311-4379-ba49-7d334a64d6b5\" (UID: \"058afa51-f311-4379-ba49-7d334a64d6b5\") " Jul 29 19:03:20.822591 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.822556 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058afa51-f311-4379-ba49-7d334a64d6b5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "058afa51-f311-4379-ba49-7d334a64d6b5" (UID: "058afa51-f311-4379-ba49-7d334a64d6b5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:03:20.824270 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.824243 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/058afa51-f311-4379-ba49-7d334a64d6b5-kube-api-access-nzd5g" (OuterVolumeSpecName: "kube-api-access-nzd5g") pod "058afa51-f311-4379-ba49-7d334a64d6b5" (UID: "058afa51-f311-4379-ba49-7d334a64d6b5"). InnerVolumeSpecName "kube-api-access-nzd5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:03:20.824270 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.824249 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "058afa51-f311-4379-ba49-7d334a64d6b5" (UID: "058afa51-f311-4379-ba49-7d334a64d6b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:03:20.824434 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.824321 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "058afa51-f311-4379-ba49-7d334a64d6b5" (UID: "058afa51-f311-4379-ba49-7d334a64d6b5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:03:20.923801 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.923765 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/058afa51-f311-4379-ba49-7d334a64d6b5-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:20.923801 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.923800 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:20.924056 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.923815 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/058afa51-f311-4379-ba49-7d334a64d6b5-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:20.924056 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:20.923828 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nzd5g\" (UniqueName: \"kubernetes.io/projected/058afa51-f311-4379-ba49-7d334a64d6b5-kube-api-access-nzd5g\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:21.630977 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.630942 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x"] Jul 29 19:03:21.631381 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631367 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058afa51-f311-4379-ba49-7d334a64d6b5" containerName="maas-api" Jul 29 19:03:21.631427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631383 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="058afa51-f311-4379-ba49-7d334a64d6b5" containerName="maas-api" Jul 29 19:03:21.631427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631393 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" Jul 29 19:03:21.631427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631399 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" Jul 29 19:03:21.631427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631405 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" Jul 29 19:03:21.631427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631410 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" Jul 29 19:03:21.631643 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631481 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" Jul 29 19:03:21.631643 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.631491 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="058afa51-f311-4379-ba49-7d334a64d6b5" containerName="maas-api" Jul 29 19:03:21.635304 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.635274 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.638616 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.638593 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1b5315ce-openshift-default-dockercfg-jdk9w\"" Jul 29 19:03:21.647008 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.646982 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x"] Jul 29 19:03:21.690398 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.690363 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" event={"ID":"058afa51-f311-4379-ba49-7d334a64d6b5","Type":"ContainerDied","Data":"d71a7badfb98c0249ec7b4d40aac2ceea38464e02f6c09e92117a05fd54abfd9"} Jul 29 19:03:21.690398 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.690396 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg" Jul 29 19:03:21.690637 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.690434 2574 scope.go:117] "RemoveContainer" containerID="5b41322d6bc143fe1a36ab407649c2ad8c7be2fca41a87f1721ecd6f6256d1da" Jul 29 19:03:21.717560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.717513 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg"] Jul 29 19:03:21.720639 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.720543 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-03561f0f-dbc78cdd-kkzmg"] Jul 29 19:03:21.730462 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.730431 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-data\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.730653 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.730635 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-podinfo\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.730786 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.730771 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxj5j\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-kube-api-access-dxj5j\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.730891 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.730879 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-socket\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.731016 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.731002 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98fa59e-585a-41d5-855b-3c28577ac1f3-istiod-ca-cert\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.731121 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.731108 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-credential-socket\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.731206 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.731195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-envoy\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.731322 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.731311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-certs\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.731441 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.731426 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-token\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832417 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832382 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-data\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832435 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-podinfo\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832471 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxj5j\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-kube-api-access-dxj5j\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-socket\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98fa59e-585a-41d5-855b-3c28577ac1f3-istiod-ca-cert\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-credential-socket\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832577 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-envoy\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832622 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-certs\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832671 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-token\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832793 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-data\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.832972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-socket\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.833185 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.832978 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-credential-socket\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.833185 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.833109 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-certs\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.833427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.833401 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98fa59e-585a-41d5-855b-3c28577ac1f3-istiod-ca-cert\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.835152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.835131 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-envoy\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.835388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.835370 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-podinfo\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.841130 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.841106 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-token\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.841332 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.841310 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxj5j\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-kube-api-access-dxj5j\") pod \"e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:21.946976 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:21.946911 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:22.087976 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.087950 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x"] Jul 29 19:03:22.090381 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:03:22.090352 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98fa59e_585a_41d5_855b_3c28577ac1f3.slice/crio-6b812440a69abfd5a50b43b77e6b7f61d5eb4a164118277d99138651d6ad3d1e WatchSource:0}: Error finding container 6b812440a69abfd5a50b43b77e6b7f61d5eb4a164118277d99138651d6ad3d1e: Status 404 returned error can't find the container with id 6b812440a69abfd5a50b43b77e6b7f61d5eb4a164118277d99138651d6ad3d1e Jul 29 19:03:22.092837 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.092796 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:22.092952 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.092882 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:22.093038 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.092954 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:22.696893 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.696853 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" event={"ID":"b98fa59e-585a-41d5-855b-3c28577ac1f3","Type":"ContainerStarted","Data":"91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab"} Jul 29 19:03:22.697388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.696913 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" event={"ID":"b98fa59e-585a-41d5-855b-3c28577ac1f3","Type":"ContainerStarted","Data":"6b812440a69abfd5a50b43b77e6b7f61d5eb4a164118277d99138651d6ad3d1e"} Jul 29 19:03:22.717558 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.717496 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podStartSLOduration=1.7174805800000001 podStartE2EDuration="1.71748058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:03:22.716390552 +0000 UTC m=+1742.429302043" watchObservedRunningTime="2026-07-29 19:03:22.71748058 +0000 UTC m=+1742.430392130" Jul 29 19:03:22.767526 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.767496 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058afa51-f311-4379-ba49-7d334a64d6b5" path="/var/lib/kubelet/pods/058afa51-f311-4379-ba49-7d334a64d6b5/volumes" Jul 29 19:03:22.947423 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:22.947377 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:23.952231 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:23.952198 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:24.282280 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.282196 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5"] Jul 29 19:03:24.282842 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.282827 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="41fd494d-f7c6-4473-b58a-7bbe08d1efae" containerName="istio-proxy" Jul 29 19:03:24.285284 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.285263 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.288447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.288428 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-1b5315ce-serving-cert\"" Jul 29 19:03:24.309110 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.309081 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5"] Jul 29 19:03:24.460194 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.460163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d82ef7-0ee3-48a0-8838-2c667287891f-openshift-service-ca\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.460417 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.460205 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrr8g\" (UniqueName: \"kubernetes.io/projected/27d82ef7-0ee3-48a0-8838-2c667287891f-kube-api-access-qrr8g\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.460417 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.460312 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-metrics-certs\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.460417 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.460362 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-maas-api-tls\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.561829 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.561738 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-maas-api-tls\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.561829 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.561791 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d82ef7-0ee3-48a0-8838-2c667287891f-openshift-service-ca\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.561829 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.561815 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrr8g\" (UniqueName: \"kubernetes.io/projected/27d82ef7-0ee3-48a0-8838-2c667287891f-kube-api-access-qrr8g\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.562111 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.561862 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-metrics-certs\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.562570 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.562539 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d82ef7-0ee3-48a0-8838-2c667287891f-openshift-service-ca\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.564436 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.564415 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-maas-api-tls\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.564577 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.564557 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-metrics-certs\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.571781 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.571760 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrr8g\" (UniqueName: \"kubernetes.io/projected/27d82ef7-0ee3-48a0-8838-2c667287891f-kube-api-access-qrr8g\") pod \"maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.595678 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.595633 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:24.712503 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.712475 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:24.713476 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.713450 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:24.734988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:24.734956 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5"] Jul 29 19:03:24.737024 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:03:24.736980 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d82ef7_0ee3_48a0_8838_2c667287891f.slice/crio-361bccaa75e075c37d4be71bf60ab0d1bc16baefc73e2d129c51912b1aa2696d WatchSource:0}: Error finding container 361bccaa75e075c37d4be71bf60ab0d1bc16baefc73e2d129c51912b1aa2696d: Status 404 returned error can't find the container with id 361bccaa75e075c37d4be71bf60ab0d1bc16baefc73e2d129c51912b1aa2696d Jul 29 19:03:25.222805 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:25.222769 2574 patch_prober.go:28] interesting pod/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 29 19:03:25.223199 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:25.222837 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:03:25.719665 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:25.719631 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" event={"ID":"27d82ef7-0ee3-48a0-8838-2c667287891f","Type":"ContainerStarted","Data":"361bccaa75e075c37d4be71bf60ab0d1bc16baefc73e2d129c51912b1aa2696d"} Jul 29 19:03:26.057854 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.057826 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:03:26.078449 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078234 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-certs\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078661 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078492 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5727fe90-77b5-4978-b890-6f6f9475edec-istio-podinfo\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078661 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078539 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-data\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078661 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078565 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-envoy\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078661 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078591 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-credential-socket\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078661 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078618 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-istio-token\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078720 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7xkd\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-kube-api-access-t7xkd\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078777 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-socket\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078808 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5727fe90-77b5-4978-b890-6f6f9475edec-istiod-ca-cert\") pod \"5727fe90-77b5-4978-b890-6f6f9475edec\" (UID: \"5727fe90-77b5-4978-b890-6f6f9475edec\") " Jul 29 19:03:26.078958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.078884 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:03:26.079165 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.079153 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.079301 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.079234 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:03:26.079694 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.079653 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:03:26.080885 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.080861 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-data" (OuterVolumeSpecName: "istio-data") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:03:26.081973 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.081949 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:03:26.082352 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.082312 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-istio-token" (OuterVolumeSpecName: "istio-token") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:03:26.083407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.082787 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5727fe90-77b5-4978-b890-6f6f9475edec-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:03:26.083407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.083144 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5727fe90-77b5-4978-b890-6f6f9475edec-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:03:26.083711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.083677 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-kube-api-access-t7xkd" (OuterVolumeSpecName: "kube-api-access-t7xkd") pod "5727fe90-77b5-4978-b890-6f6f9475edec" (UID: "5727fe90-77b5-4978-b890-6f6f9475edec"). InnerVolumeSpecName "kube-api-access-t7xkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:03:26.179747 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179697 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.179747 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179738 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7xkd\" (UniqueName: \"kubernetes.io/projected/5727fe90-77b5-4978-b890-6f6f9475edec-kube-api-access-t7xkd\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.179747 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179755 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.180080 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179769 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5727fe90-77b5-4978-b890-6f6f9475edec-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.180080 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179782 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5727fe90-77b5-4978-b890-6f6f9475edec-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.180080 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179795 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.180080 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179806 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.180080 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.179818 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5727fe90-77b5-4978-b890-6f6f9475edec-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:26.726451 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.726412 2574 generic.go:356] "Generic (PLEG): container finished" podID="5727fe90-77b5-4978-b890-6f6f9475edec" containerID="e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3" exitCode=0 Jul 29 19:03:26.726905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.726469 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" Jul 29 19:03:26.726905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.726493 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" event={"ID":"5727fe90-77b5-4978-b890-6f6f9475edec","Type":"ContainerDied","Data":"e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3"} Jul 29 19:03:26.726905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.726548 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w" event={"ID":"5727fe90-77b5-4978-b890-6f6f9475edec","Type":"ContainerDied","Data":"4b7a74493da77eb11cb458b3bce49b60866a3f34741e043ed9f8481570ff3584"} Jul 29 19:03:26.726905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.726571 2574 scope.go:117] "RemoveContainer" containerID="e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3" Jul 29 19:03:26.728207 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.728182 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" event={"ID":"27d82ef7-0ee3-48a0-8838-2c667287891f","Type":"ContainerStarted","Data":"f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a"} Jul 29 19:03:26.728445 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.728430 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:26.737699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.737683 2574 scope.go:117] "RemoveContainer" containerID="86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b" Jul 29 19:03:26.745637 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.745619 2574 scope.go:117] "RemoveContainer" containerID="e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3" Jul 29 19:03:26.745862 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:03:26.745844 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3\": container with ID starting with e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3 not found: ID does not exist" containerID="e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3" Jul 29 19:03:26.745910 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.745870 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3"} err="failed to get container status \"e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3\": rpc error: code = NotFound desc = could not find container \"e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3\": container with ID starting with e8ccf46eab081bd566a88d8326d6984aaf3a6697a68a524d6eac85b8ae8a7db3 not found: ID does not exist" Jul 29 19:03:26.745910 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.745888 2574 scope.go:117] "RemoveContainer" containerID="86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b" Jul 29 19:03:26.746124 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:03:26.746109 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b\": container with ID starting with 86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b not found: ID does not exist" containerID="86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b" Jul 29 19:03:26.746164 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.746128 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b"} err="failed to get container status \"86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b\": rpc error: code = NotFound desc = could not find container \"86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b\": container with ID starting with 86699779be8ccf5826f3cadf8db26c5f72932480b98814e83eceb83c9842e46b not found: ID does not exist" Jul 29 19:03:26.751950 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.751888 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" podStartSLOduration=1.693879446 podStartE2EDuration="2.751875204s" totalImagesPullingTime="1.057995758s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:03:26.751619715 +0000 UTC m=+1746.464531205" watchObservedRunningTime="2026-07-29 19:03:26.751875204 +0000 UTC m=+1746.464786694" Jul 29 19:03:26.767637 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.767612 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w"] Jul 29 19:03:26.771249 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:26.771227 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-03561f0f-gw-openshift-default-7b97554484-ql88w"] Jul 29 19:03:28.766579 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:28.766543 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" path="/var/lib/kubelet/pods/5727fe90-77b5-4978-b890-6f6f9475edec/volumes" Jul 29 19:03:32.739784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:32.739748 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:38.785457 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:38.785426 2574 generic.go:356] "Generic (PLEG): container finished" podID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerID="91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab" exitCode=0 Jul 29 19:03:38.785895 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:38.785504 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" event={"ID":"b98fa59e-585a-41d5-855b-3c28577ac1f3","Type":"ContainerDied","Data":"91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab"} Jul 29 19:03:38.785969 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:38.785897 2574 scope.go:117] "RemoveContainer" containerID="91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab" Jul 29 19:03:38.786905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:38.786871 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:38.787214 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:38.786965 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:38.787214 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:38.786996 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:39.712822 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:39.712775 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:39.797559 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:39.797523 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" event={"ID":"b98fa59e-585a-41d5-855b-3c28577ac1f3","Type":"ContainerStarted","Data":"789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e"} Jul 29 19:03:39.947566 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:39.947536 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:39.948860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:39.948835 2574 patch_prober.go:28] interesting pod/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 29 19:03:39.948980 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:39.948880 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 29 19:03:40.947338 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:40.947304 2574 patch_prober.go:28] interesting pod/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 29 19:03:40.947717 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:40.947372 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 29 19:03:41.947326 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:41.947289 2574 patch_prober.go:28] interesting pod/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 29 19:03:41.947501 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:41.947341 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 29 19:03:42.947790 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:42.947752 2574 patch_prober.go:28] interesting pod/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Jul 29 19:03:42.948251 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:42.947819 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Jul 29 19:03:43.951226 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:43.951195 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:43.951695 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:43.951583 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:43.952205 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:43.952185 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:03:54.919698 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:54.919655 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5"] Jul 29 19:03:54.920225 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:54.919977 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" podUID="27d82ef7-0ee3-48a0-8838-2c667287891f" containerName="maas-api" containerID="cri-o://f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a" gracePeriod=30 Jul 29 19:03:55.171706 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.171639 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:55.252129 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.252093 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-metrics-certs\") pod \"27d82ef7-0ee3-48a0-8838-2c667287891f\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " Jul 29 19:03:55.252312 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.252166 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrr8g\" (UniqueName: \"kubernetes.io/projected/27d82ef7-0ee3-48a0-8838-2c667287891f-kube-api-access-qrr8g\") pod \"27d82ef7-0ee3-48a0-8838-2c667287891f\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " Jul 29 19:03:55.252312 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.252240 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d82ef7-0ee3-48a0-8838-2c667287891f-openshift-service-ca\") pod \"27d82ef7-0ee3-48a0-8838-2c667287891f\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " Jul 29 19:03:55.252408 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.252322 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-maas-api-tls\") pod \"27d82ef7-0ee3-48a0-8838-2c667287891f\" (UID: \"27d82ef7-0ee3-48a0-8838-2c667287891f\") " Jul 29 19:03:55.252617 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.252581 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d82ef7-0ee3-48a0-8838-2c667287891f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "27d82ef7-0ee3-48a0-8838-2c667287891f" (UID: "27d82ef7-0ee3-48a0-8838-2c667287891f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:03:55.254470 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.254439 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "27d82ef7-0ee3-48a0-8838-2c667287891f" (UID: "27d82ef7-0ee3-48a0-8838-2c667287891f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:03:55.254565 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.254489 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "27d82ef7-0ee3-48a0-8838-2c667287891f" (UID: "27d82ef7-0ee3-48a0-8838-2c667287891f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:03:55.254565 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.254494 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d82ef7-0ee3-48a0-8838-2c667287891f-kube-api-access-qrr8g" (OuterVolumeSpecName: "kube-api-access-qrr8g") pod "27d82ef7-0ee3-48a0-8838-2c667287891f" (UID: "27d82ef7-0ee3-48a0-8838-2c667287891f"). InnerVolumeSpecName "kube-api-access-qrr8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:03:55.353642 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.353609 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrr8g\" (UniqueName: \"kubernetes.io/projected/27d82ef7-0ee3-48a0-8838-2c667287891f-kube-api-access-qrr8g\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:55.353642 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.353638 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/27d82ef7-0ee3-48a0-8838-2c667287891f-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:55.353642 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.353647 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:55.353897 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.353656 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27d82ef7-0ee3-48a0-8838-2c667287891f-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:03:55.886264 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.886222 2574 generic.go:356] "Generic (PLEG): container finished" podID="27d82ef7-0ee3-48a0-8838-2c667287891f" containerID="f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a" exitCode=0 Jul 29 19:03:55.886448 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.886273 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" Jul 29 19:03:55.886448 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.886278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" event={"ID":"27d82ef7-0ee3-48a0-8838-2c667287891f","Type":"ContainerDied","Data":"f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a"} Jul 29 19:03:55.886448 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.886343 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5" event={"ID":"27d82ef7-0ee3-48a0-8838-2c667287891f","Type":"ContainerDied","Data":"361bccaa75e075c37d4be71bf60ab0d1bc16baefc73e2d129c51912b1aa2696d"} Jul 29 19:03:55.886448 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.886372 2574 scope.go:117] "RemoveContainer" containerID="f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a" Jul 29 19:03:55.905209 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.905186 2574 scope.go:117] "RemoveContainer" containerID="f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a" Jul 29 19:03:55.905546 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:03:55.905515 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a\": container with ID starting with f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a not found: ID does not exist" containerID="f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a" Jul 29 19:03:55.905650 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.905559 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a"} err="failed to get container status \"f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a\": rpc error: code = NotFound desc = could not find container \"f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a\": container with ID starting with f6ac63eed7dc2fb9a9e2461aa01b328bbbee7b33b155ff8389f3cd8e76c39c3a not found: ID does not exist" Jul 29 19:03:55.918654 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.918620 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5"] Jul 29 19:03:55.924697 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:55.924668 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-1b5315ce-69ffcc546c-gqwx5"] Jul 29 19:03:56.429185 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:56.429151 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x"] Jul 29 19:03:56.429455 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:56.429419 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" containerID="cri-o://789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e" gracePeriod=30 Jul 29 19:03:56.768078 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:56.767991 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27d82ef7-0ee3-48a0-8838-2c667287891f" path="/var/lib/kubelet/pods/27d82ef7-0ee3-48a0-8838-2c667287891f/volumes" Jul 29 19:03:57.225187 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225145 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc"] Jul 29 19:03:57.225794 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225760 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" Jul 29 19:03:57.225794 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225787 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" Jul 29 19:03:57.225794 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225797 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" Jul 29 19:03:57.226051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225805 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" Jul 29 19:03:57.226051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225824 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27d82ef7-0ee3-48a0-8838-2c667287891f" containerName="maas-api" Jul 29 19:03:57.226051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225831 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="27d82ef7-0ee3-48a0-8838-2c667287891f" containerName="maas-api" Jul 29 19:03:57.226051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225900 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="27d82ef7-0ee3-48a0-8838-2c667287891f" containerName="maas-api" Jul 29 19:03:57.226051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225909 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" Jul 29 19:03:57.226051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.225944 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="5727fe90-77b5-4978-b890-6f6f9475edec" containerName="istio-proxy" Jul 29 19:03:57.229247 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.229227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.232102 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.232080 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-77117817-openshift-default-dockercfg-4xrcv\"" Jul 29 19:03:57.241245 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.241221 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc"] Jul 29 19:03:57.374314 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istiod-ca-cert\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374494 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374328 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-data\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374494 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-token\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374494 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374443 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-envoy\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374502 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-socket\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374559 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-credential-socket\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374577 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6b2s\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-kube-api-access-z6b2s\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374696 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374609 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-certs\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.374696 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.374623 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-podinfo\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476105 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476016 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-certs\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476105 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476057 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-podinfo\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476122 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istiod-ca-cert\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476164 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-data\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476186 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-token\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476249 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-envoy\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476298 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-socket\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-credential-socket\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476359 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6b2s\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-kube-api-access-z6b2s\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476794 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476536 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-certs\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476794 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476639 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-data\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476956 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476897 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-credential-socket\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.476956 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476943 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istiod-ca-cert\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.477066 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.476982 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-socket\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.478762 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.478733 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-envoy\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.478855 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.478832 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-podinfo\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.484699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.484670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-token\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.484839 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.484742 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6b2s\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-kube-api-access-z6b2s\") pod \"e2e-derive-77117817-openshift-default-748d556d96-jjgnc\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.539783 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.539743 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:57.710550 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.710519 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc"] Jul 29 19:03:57.711993 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:03:57.711963 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3e5ee6_4a45_4b8b_b741_a9fc680c370b.slice/crio-ad26d20c9a2bf158d81ba1496f435e4deda1bc04f9640904f12e7e603492570f WatchSource:0}: Error finding container ad26d20c9a2bf158d81ba1496f435e4deda1bc04f9640904f12e7e603492570f: Status 404 returned error can't find the container with id ad26d20c9a2bf158d81ba1496f435e4deda1bc04f9640904f12e7e603492570f Jul 29 19:03:57.714662 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.714623 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:57.714780 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.714761 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:57.714825 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.714810 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:03:57.900843 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.900801 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" event={"ID":"df3e5ee6-4a45-4b8b-b741-a9fc680c370b","Type":"ContainerStarted","Data":"9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82"} Jul 29 19:03:57.901052 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.900863 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" event={"ID":"df3e5ee6-4a45-4b8b-b741-a9fc680c370b","Type":"ContainerStarted","Data":"ad26d20c9a2bf158d81ba1496f435e4deda1bc04f9640904f12e7e603492570f"} Jul 29 19:03:57.921348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:57.921293 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podStartSLOduration=0.921272669 podStartE2EDuration="921.272669ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:03:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:03:57.920031064 +0000 UTC m=+1777.632942554" watchObservedRunningTime="2026-07-29 19:03:57.921272669 +0000 UTC m=+1777.634184160" Jul 29 19:03:58.540226 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:58.540190 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:58.545527 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:58.545502 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:58.906121 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:58.906024 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:58.907167 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:58.907131 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:03:58.954179 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:58.954138 2574 patch_prober.go:28] interesting pod/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:03:58.954347 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:58.954210 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:03:59.381942 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.381888 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl"] Jul 29 19:03:59.384659 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.384635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.387533 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.387511 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-77117817-serving-cert\"" Jul 29 19:03:59.401823 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.401794 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl"] Jul 29 19:03:59.496476 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.496430 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-metrics-certs\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.496687 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.496500 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-maas-api-tls\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.496687 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.496536 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a2e78e-3184-4493-a0a9-d467ae1ddd40-openshift-service-ca\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.496687 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.496640 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krww5\" (UniqueName: \"kubernetes.io/projected/04a2e78e-3184-4493-a0a9-d467ae1ddd40-kube-api-access-krww5\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.597213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.597180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a2e78e-3184-4493-a0a9-d467ae1ddd40-openshift-service-ca\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.597732 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.597258 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krww5\" (UniqueName: \"kubernetes.io/projected/04a2e78e-3184-4493-a0a9-d467ae1ddd40-kube-api-access-krww5\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.597732 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.597294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-metrics-certs\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.597732 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.597379 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-maas-api-tls\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.598100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.598071 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a2e78e-3184-4493-a0a9-d467ae1ddd40-openshift-service-ca\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.602400 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.600777 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-metrics-certs\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.602400 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.601797 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-maas-api-tls\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.607707 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.607675 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-krww5\" (UniqueName: \"kubernetes.io/projected/04a2e78e-3184-4493-a0a9-d467ae1ddd40-kube-api-access-krww5\") pod \"maas-api-e2e-derive-77117817-5fc44d6-dv6vl\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.698305 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.698266 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:03:59.857402 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.857373 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl"] Jul 29 19:03:59.859210 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:03:59.859179 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a2e78e_3184_4493_a0a9_d467ae1ddd40.slice/crio-f245d812894690832297a82b46353dc566f6e4a0e2f068002ad4ae6276e5899b WatchSource:0}: Error finding container f245d812894690832297a82b46353dc566f6e4a0e2f068002ad4ae6276e5899b: Status 404 returned error can't find the container with id f245d812894690832297a82b46353dc566f6e4a0e2f068002ad4ae6276e5899b Jul 29 19:03:59.919303 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:03:59.919082 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" event={"ID":"04a2e78e-3184-4493-a0a9-d467ae1ddd40","Type":"ContainerStarted","Data":"f245d812894690832297a82b46353dc566f6e4a0e2f068002ad4ae6276e5899b"} Jul 29 19:04:00.925759 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:00.925721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" event={"ID":"04a2e78e-3184-4493-a0a9-d467ae1ddd40","Type":"ContainerStarted","Data":"013b7dec70f04ea736864c2719e2ad5ed182a2f8b50b497b25beffd8dc400708"} Jul 29 19:04:00.926278 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:00.925785 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:04:00.946400 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:00.946353 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" podStartSLOduration=1.606840914 podStartE2EDuration="1.946336249s" totalImagesPullingTime="339.495335ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:03:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:04:00.944598521 +0000 UTC m=+1780.657510027" watchObservedRunningTime="2026-07-29 19:04:00.946336249 +0000 UTC m=+1780.659247746" Jul 29 19:04:01.689141 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.689074 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:04:01.820270 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820235 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-podinfo\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98fa59e-585a-41d5-855b-3c28577ac1f3-istiod-ca-cert\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820335 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxj5j\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-kube-api-access-dxj5j\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820357 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-envoy\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820375 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-socket\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820411 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-token\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820461 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-certs\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820526 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-data\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820553 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-credential-socket\") pod \"b98fa59e-585a-41d5-855b-3c28577ac1f3\" (UID: \"b98fa59e-585a-41d5-855b-3c28577ac1f3\") " Jul 29 19:04:01.820731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820706 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:01.820997 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:01.820997 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820895 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98fa59e-585a-41d5-855b-3c28577ac1f3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:04:01.820997 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.820900 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:01.821279 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.821254 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.821403 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.821283 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.822060 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.822026 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-data" (OuterVolumeSpecName: "istio-data") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:01.822743 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.822713 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-token" (OuterVolumeSpecName: "istio-token") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:04:01.822824 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.822759 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:04:01.822886 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.822869 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-kube-api-access-dxj5j" (OuterVolumeSpecName: "kube-api-access-dxj5j") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "kube-api-access-dxj5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:04:01.823027 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.823010 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b98fa59e-585a-41d5-855b-3c28577ac1f3" (UID: "b98fa59e-585a-41d5-855b-3c28577ac1f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923132 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923165 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b98fa59e-585a-41d5-855b-3c28577ac1f3-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923185 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxj5j\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-kube-api-access-dxj5j\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923202 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923216 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923229 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.923271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.923247 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b98fa59e-585a-41d5-855b-3c28577ac1f3-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:01.932050 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.932012 2574 generic.go:356] "Generic (PLEG): container finished" podID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerID="789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e" exitCode=0 Jul 29 19:04:01.932477 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.932061 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" Jul 29 19:04:01.932477 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.932089 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" event={"ID":"b98fa59e-585a-41d5-855b-3c28577ac1f3","Type":"ContainerDied","Data":"789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e"} Jul 29 19:04:01.932477 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.932145 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x" event={"ID":"b98fa59e-585a-41d5-855b-3c28577ac1f3","Type":"ContainerDied","Data":"6b812440a69abfd5a50b43b77e6b7f61d5eb4a164118277d99138651d6ad3d1e"} Jul 29 19:04:01.932477 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.932167 2574 scope.go:117] "RemoveContainer" containerID="789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e" Jul 29 19:04:01.942398 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.942377 2574 scope.go:117] "RemoveContainer" containerID="91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab" Jul 29 19:04:01.951618 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.951601 2574 scope.go:117] "RemoveContainer" containerID="789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e" Jul 29 19:04:01.951873 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:04:01.951854 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e\": container with ID starting with 789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e not found: ID does not exist" containerID="789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e" Jul 29 19:04:01.951988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.951879 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e"} err="failed to get container status \"789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e\": rpc error: code = NotFound desc = could not find container \"789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e\": container with ID starting with 789b5763988686e4f22d8d0084d616d3ddb88e3c291dc32d194236086954577e not found: ID does not exist" Jul 29 19:04:01.951988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.951897 2574 scope.go:117] "RemoveContainer" containerID="91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab" Jul 29 19:04:01.952188 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:04:01.952168 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab\": container with ID starting with 91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab not found: ID does not exist" containerID="91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab" Jul 29 19:04:01.952249 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.952193 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab"} err="failed to get container status \"91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab\": rpc error: code = NotFound desc = could not find container \"91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab\": container with ID starting with 91a45719cd8ae66971422a38c4115e34a438def4140471fdf1407041105d4eab not found: ID does not exist" Jul 29 19:04:01.956798 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.956772 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x"] Jul 29 19:04:01.960530 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:01.960508 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1b5315ce-openshift-default-5d6c5cf894-j4m9x"] Jul 29 19:04:02.768152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:02.768118 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" path="/var/lib/kubelet/pods/b98fa59e-585a-41d5-855b-3c28577ac1f3/volumes" Jul 29 19:04:06.938190 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:06.938158 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:04:12.987683 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:12.987649 2574 generic.go:356] "Generic (PLEG): container finished" podID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerID="9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82" exitCode=0 Jul 29 19:04:12.988132 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:12.987732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" event={"ID":"df3e5ee6-4a45-4b8b-b741-a9fc680c370b","Type":"ContainerDied","Data":"9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82"} Jul 29 19:04:12.988236 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:12.988219 2574 scope.go:117] "RemoveContainer" containerID="9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82" Jul 29 19:04:12.989381 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:12.989353 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:04:12.989455 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:12.989441 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:04:12.989491 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:12.989477 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:04:13.906494 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:13.906452 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:13.998034 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:13.997980 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" event={"ID":"df3e5ee6-4a45-4b8b-b741-a9fc680c370b","Type":"ContainerStarted","Data":"56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171"} Jul 29 19:04:14.540972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:14.540906 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:14.541958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:14.541910 2574 patch_prober.go:28] interesting pod/e2e-derive-77117817-openshift-default-748d556d96-jjgnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 19:04:14.542078 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:14.541993 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 19:04:15.540748 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:15.540711 2574 patch_prober.go:28] interesting pod/e2e-derive-77117817-openshift-default-748d556d96-jjgnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 19:04:15.541152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:15.540766 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 19:04:16.540783 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:16.540748 2574 patch_prober.go:28] interesting pod/e2e-derive-77117817-openshift-default-748d556d96-jjgnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 19:04:16.541194 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:16.540833 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 19:04:17.541082 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:17.541041 2574 patch_prober.go:28] interesting pod/e2e-derive-77117817-openshift-default-748d556d96-jjgnc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Jul 29 19:04:17.541482 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:17.541113 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Jul 29 19:04:18.544479 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:18.544446 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:18.544979 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:18.544960 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:18.545601 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:18.545584 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:29.965887 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:29.965846 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl"] Jul 29 19:04:29.966497 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:29.966464 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" podUID="04a2e78e-3184-4493-a0a9-d467ae1ddd40" containerName="maas-api" containerID="cri-o://013b7dec70f04ea736864c2719e2ad5ed182a2f8b50b497b25beffd8dc400708" gracePeriod=30 Jul 29 19:04:30.086660 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.086624 2574 generic.go:356] "Generic (PLEG): container finished" podID="04a2e78e-3184-4493-a0a9-d467ae1ddd40" containerID="013b7dec70f04ea736864c2719e2ad5ed182a2f8b50b497b25beffd8dc400708" exitCode=0 Jul 29 19:04:30.086796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.086666 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" event={"ID":"04a2e78e-3184-4493-a0a9-d467ae1ddd40","Type":"ContainerDied","Data":"013b7dec70f04ea736864c2719e2ad5ed182a2f8b50b497b25beffd8dc400708"} Jul 29 19:04:30.230342 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.230256 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:04:30.306848 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.306795 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a2e78e-3184-4493-a0a9-d467ae1ddd40-openshift-service-ca\") pod \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " Jul 29 19:04:30.307197 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.306855 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-metrics-certs\") pod \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " Jul 29 19:04:30.307197 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.306889 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-maas-api-tls\") pod \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " Jul 29 19:04:30.307197 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.306975 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-krww5\" (UniqueName: \"kubernetes.io/projected/04a2e78e-3184-4493-a0a9-d467ae1ddd40-kube-api-access-krww5\") pod \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\" (UID: \"04a2e78e-3184-4493-a0a9-d467ae1ddd40\") " Jul 29 19:04:30.307851 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.307810 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a2e78e-3184-4493-a0a9-d467ae1ddd40-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04a2e78e-3184-4493-a0a9-d467ae1ddd40" (UID: "04a2e78e-3184-4493-a0a9-d467ae1ddd40"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:04:30.309964 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.309902 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a2e78e-3184-4493-a0a9-d467ae1ddd40-kube-api-access-krww5" (OuterVolumeSpecName: "kube-api-access-krww5") pod "04a2e78e-3184-4493-a0a9-d467ae1ddd40" (UID: "04a2e78e-3184-4493-a0a9-d467ae1ddd40"). InnerVolumeSpecName "kube-api-access-krww5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:04:30.310207 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.310173 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04a2e78e-3184-4493-a0a9-d467ae1ddd40" (UID: "04a2e78e-3184-4493-a0a9-d467ae1ddd40"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:04:30.310207 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.310185 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "04a2e78e-3184-4493-a0a9-d467ae1ddd40" (UID: "04a2e78e-3184-4493-a0a9-d467ae1ddd40"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:04:30.341863 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.341776 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc"] Jul 29 19:04:30.342176 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.342146 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" containerID="cri-o://56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171" gracePeriod=30 Jul 29 19:04:30.408069 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.408028 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a2e78e-3184-4493-a0a9-d467ae1ddd40-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:30.408069 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.408063 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:30.408069 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.408073 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a2e78e-3184-4493-a0a9-d467ae1ddd40-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:30.408318 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:30.408081 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krww5\" (UniqueName: \"kubernetes.io/projected/04a2e78e-3184-4493-a0a9-d467ae1ddd40-kube-api-access-krww5\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:31.093834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:31.093720 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" event={"ID":"04a2e78e-3184-4493-a0a9-d467ae1ddd40","Type":"ContainerDied","Data":"f245d812894690832297a82b46353dc566f6e4a0e2f068002ad4ae6276e5899b"} Jul 29 19:04:31.093834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:31.093782 2574 scope.go:117] "RemoveContainer" containerID="013b7dec70f04ea736864c2719e2ad5ed182a2f8b50b497b25beffd8dc400708" Jul 29 19:04:31.093834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:31.093731 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl" Jul 29 19:04:31.129978 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:31.129940 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl"] Jul 29 19:04:31.142824 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:31.142790 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-77117817-5fc44d6-dv6vl"] Jul 29 19:04:32.767417 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:32.767385 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a2e78e-3184-4493-a0a9-d467ae1ddd40" path="/var/lib/kubelet/pods/04a2e78e-3184-4493-a0a9-d467ae1ddd40/volumes" Jul 29 19:04:33.547321 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:33.547279 2574 patch_prober.go:28] interesting pod/e2e-derive-77117817-openshift-default-748d556d96-jjgnc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:04:33.547487 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:33.547342 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:04:35.624664 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.624640 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:35.760493 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760460 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-envoy\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760504 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istiod-ca-cert\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760561 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-credential-socket\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760591 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-certs\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760880 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760737 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-podinfo\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760880 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760784 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-socket\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760880 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760810 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6b2s\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-kube-api-access-z6b2s\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.760880 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760870 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-data\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.761124 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760902 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:35.761124 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760948 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-token\") pod \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\" (UID: \"df3e5ee6-4a45-4b8b-b741-a9fc680c370b\") " Jul 29 19:04:35.761368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.760944 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:35.761702 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.761043 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:04:35.761702 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.761155 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:35.761702 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.761375 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.761879 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.761705 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-data" (OuterVolumeSpecName: "istio-data") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:35.762999 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.762970 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:04:35.763120 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.763099 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-kube-api-access-z6b2s" (OuterVolumeSpecName: "kube-api-access-z6b2s") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "kube-api-access-z6b2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:04:35.763702 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.763684 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:04:35.763782 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.763712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-token" (OuterVolumeSpecName: "istio-token") pod "df3e5ee6-4a45-4b8b-b741-a9fc680c370b" (UID: "df3e5ee6-4a45-4b8b-b741-a9fc680c370b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862860 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862885 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862896 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862907 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6b2s\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-kube-api-access-z6b2s\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862948 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862959 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.862965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862967 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:35.863303 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:35.862974 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df3e5ee6-4a45-4b8b-b741-a9fc680c370b-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:04:36.119322 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.119238 2574 generic.go:356] "Generic (PLEG): container finished" podID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerID="56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171" exitCode=0 Jul 29 19:04:36.119322 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.119307 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" Jul 29 19:04:36.119489 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.119328 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" event={"ID":"df3e5ee6-4a45-4b8b-b741-a9fc680c370b","Type":"ContainerDied","Data":"56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171"} Jul 29 19:04:36.119489 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.119417 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc" event={"ID":"df3e5ee6-4a45-4b8b-b741-a9fc680c370b","Type":"ContainerDied","Data":"ad26d20c9a2bf158d81ba1496f435e4deda1bc04f9640904f12e7e603492570f"} Jul 29 19:04:36.119489 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.119448 2574 scope.go:117] "RemoveContainer" containerID="56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171" Jul 29 19:04:36.128848 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.128814 2574 scope.go:117] "RemoveContainer" containerID="9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82" Jul 29 19:04:36.137085 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.137066 2574 scope.go:117] "RemoveContainer" containerID="56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171" Jul 29 19:04:36.137340 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:04:36.137320 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171\": container with ID starting with 56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171 not found: ID does not exist" containerID="56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171" Jul 29 19:04:36.137384 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.137348 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171"} err="failed to get container status \"56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171\": rpc error: code = NotFound desc = could not find container \"56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171\": container with ID starting with 56c80ae1f9829aff0ada180576fea62236baf678ac4c48c373bbe46fd7cf5171 not found: ID does not exist" Jul 29 19:04:36.137384 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.137363 2574 scope.go:117] "RemoveContainer" containerID="9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82" Jul 29 19:04:36.137573 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:04:36.137557 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82\": container with ID starting with 9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82 not found: ID does not exist" containerID="9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82" Jul 29 19:04:36.137608 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.137577 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82"} err="failed to get container status \"9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82\": rpc error: code = NotFound desc = could not find container \"9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82\": container with ID starting with 9124f123ba8c923819395abd5487e034189fc9369d5ab216f2c100c6fa8f5d82 not found: ID does not exist" Jul 29 19:04:36.142533 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.142510 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc"] Jul 29 19:04:36.146273 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.146249 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-77117817-openshift-default-748d556d96-jjgnc"] Jul 29 19:04:36.767754 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:04:36.767723 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" path="/var/lib/kubelet/pods/df3e5ee6-4a45-4b8b-b741-a9fc680c370b/volumes" Jul 29 19:06:14.012357 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012279 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z"] Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012707 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012724 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012733 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012739 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012749 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012755 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012761 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012766 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012776 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a2e78e-3184-4493-a0a9-d467ae1ddd40" containerName="maas-api" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012781 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a2e78e-3184-4493-a0a9-d467ae1ddd40" containerName="maas-api" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012854 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" Jul 29 19:06:14.012860 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012863 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" Jul 29 19:06:14.013476 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012870 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="b98fa59e-585a-41d5-855b-3c28577ac1f3" containerName="istio-proxy" Jul 29 19:06:14.013476 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012876 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a2e78e-3184-4493-a0a9-d467ae1ddd40" containerName="maas-api" Jul 29 19:06:14.013476 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.012882 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3e5ee6-4a45-4b8b-b741-a9fc680c370b" containerName="istio-proxy" Jul 29 19:06:14.014979 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.014961 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.017623 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.017602 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b5792115-openshift-default-dockercfg-tg6kq\"" Jul 29 19:06:14.043945 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.043877 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z"] Jul 29 19:06:14.071154 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071118 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-podinfo\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071322 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071192 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-data\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071322 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071276 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-certs\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071399 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071320 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e06ebf0-87ee-4013-976d-28170b8215f5-istiod-ca-cert\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071399 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dwh\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-kube-api-access-49dwh\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071471 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071397 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-socket\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071471 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-envoy\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071471 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071457 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-token\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.071592 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.071480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-credential-socket\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172097 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172061 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e06ebf0-87ee-4013-976d-28170b8215f5-istiod-ca-cert\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172097 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dwh\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-kube-api-access-49dwh\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172120 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-socket\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172150 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-envoy\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172180 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-token\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172201 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-credential-socket\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172246 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-podinfo\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172443 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-data\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-certs\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172558 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-socket\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172584 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-credential-socket\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-certs\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.172850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.172826 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-data\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.173025 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.173001 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e06ebf0-87ee-4013-976d-28170b8215f5-istiod-ca-cert\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.174733 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.174713 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-envoy\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.174866 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.174835 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-podinfo\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.182028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.182003 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-token\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.182333 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.182305 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dwh\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-kube-api-access-49dwh\") pod \"e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.325195 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.325104 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:14.467963 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.467936 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z"] Jul 29 19:06:14.470072 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:06:14.470047 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e06ebf0_87ee_4013_976d_28170b8215f5.slice/crio-8d35161b758844298ee3a7ba69e4fd0058c574fbba347a1865de2b23308936ea WatchSource:0}: Error finding container 8d35161b758844298ee3a7ba69e4fd0058c574fbba347a1865de2b23308936ea: Status 404 returned error can't find the container with id 8d35161b758844298ee3a7ba69e4fd0058c574fbba347a1865de2b23308936ea Jul 29 19:06:14.472338 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.472297 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:06:14.472415 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.472384 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:06:14.472454 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.472426 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:06:14.583464 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.583376 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" event={"ID":"8e06ebf0-87ee-4013-976d-28170b8215f5","Type":"ContainerStarted","Data":"58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e"} Jul 29 19:06:14.583464 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:14.583440 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" event={"ID":"8e06ebf0-87ee-4013-976d-28170b8215f5","Type":"ContainerStarted","Data":"8d35161b758844298ee3a7ba69e4fd0058c574fbba347a1865de2b23308936ea"} Jul 29 19:06:15.326220 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:15.326170 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:15.331308 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:15.331281 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:15.353268 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:15.353211 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podStartSLOduration=2.353193979 podStartE2EDuration="2.353193979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:06:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:14.61438879 +0000 UTC m=+1914.327300308" watchObservedRunningTime="2026-07-29 19:06:15.353193979 +0000 UTC m=+1915.066105883" Jul 29 19:06:15.590055 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:15.589962 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:15.590961 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:15.590911 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:17.087768 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.087734 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw"] Jul 29 19:06:17.090366 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.090347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.094670 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.094646 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b5792115-serving-cert\"" Jul 29 19:06:17.098076 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.098049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-openshift-service-ca\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.098217 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.098090 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-maas-api-tls\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.098217 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.098116 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-metrics-certs\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.098304 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.098223 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vvw4\" (UniqueName: \"kubernetes.io/projected/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-kube-api-access-4vvw4\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.105800 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.105768 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw"] Jul 29 19:06:17.199443 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.199413 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-openshift-service-ca\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.199641 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.199451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-maas-api-tls\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.199641 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.199467 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-metrics-certs\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.199641 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.199501 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vvw4\" (UniqueName: \"kubernetes.io/projected/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-kube-api-access-4vvw4\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.200117 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.200093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-openshift-service-ca\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.202100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.202081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-metrics-certs\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.202180 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.202122 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-maas-api-tls\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.208443 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.208420 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vvw4\" (UniqueName: \"kubernetes.io/projected/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-kube-api-access-4vvw4\") pod \"maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.404183 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.404082 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:17.551793 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.551758 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw"] Jul 29 19:06:17.552881 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:06:17.552854 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65b9ae53_5c1c_4aad_a879_8ad8270c5a81.slice/crio-b363f17fd82d7bf2c9a76c313bb3f85f1eca4b9c0462188851f8b15b44b742f0 WatchSource:0}: Error finding container b363f17fd82d7bf2c9a76c313bb3f85f1eca4b9c0462188851f8b15b44b742f0: Status 404 returned error can't find the container with id b363f17fd82d7bf2c9a76c313bb3f85f1eca4b9c0462188851f8b15b44b742f0 Jul 29 19:06:17.600962 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:17.600900 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" event={"ID":"65b9ae53-5c1c-4aad-a879-8ad8270c5a81","Type":"ContainerStarted","Data":"b363f17fd82d7bf2c9a76c313bb3f85f1eca4b9c0462188851f8b15b44b742f0"} Jul 29 19:06:19.612774 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:19.612732 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" event={"ID":"65b9ae53-5c1c-4aad-a879-8ad8270c5a81","Type":"ContainerStarted","Data":"a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254"} Jul 29 19:06:19.613290 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:19.612864 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:19.633420 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:19.633369 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" podStartSLOduration=1.6132345319999999 podStartE2EDuration="2.633350034s" totalImagesPullingTime="1.020115502s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:06:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:06:19.631200577 +0000 UTC m=+1919.344112068" watchObservedRunningTime="2026-07-29 19:06:19.633350034 +0000 UTC m=+1919.346261525" Jul 29 19:06:25.622382 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:25.622344 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:06:30.591771 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.591737 2574 patch_prober.go:28] interesting pod/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 19:06:30.592300 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.591796 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 19:06:30.671573 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.671532 2574 generic.go:356] "Generic (PLEG): container finished" podID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerID="58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e" exitCode=0 Jul 29 19:06:30.671732 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.671599 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" event={"ID":"8e06ebf0-87ee-4013-976d-28170b8215f5","Type":"ContainerDied","Data":"58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e"} Jul 29 19:06:30.672044 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.672027 2574 scope.go:117] "RemoveContainer" containerID="58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e" Jul 29 19:06:30.673161 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.673126 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:06:30.673228 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.673215 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:06:30.673278 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:30.673262 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:06:31.679009 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:31.678967 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" event={"ID":"8e06ebf0-87ee-4013-976d-28170b8215f5","Type":"ContainerStarted","Data":"8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da"} Jul 29 19:06:32.325878 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:32.325847 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:32.326072 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:32.326042 2574 patch_prober.go:28] interesting pod/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 19:06:32.326121 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:32.326087 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 19:06:33.326058 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:33.326010 2574 patch_prober.go:28] interesting pod/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 19:06:33.326524 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:33.326111 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 19:06:34.326384 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:34.326347 2574 patch_prober.go:28] interesting pod/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 19:06:34.326777 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:34.326417 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 19:06:35.325806 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:35.325769 2574 patch_prober.go:28] interesting pod/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Jul 29 19:06:35.326020 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:35.325824 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Jul 29 19:06:36.329369 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:36.329339 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:36.329821 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:36.329605 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:36.330260 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:36.330243 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:06:59.856797 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:59.856759 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw"] Jul 29 19:06:59.858147 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:06:59.856998 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" podUID="65b9ae53-5c1c-4aad-a879-8ad8270c5a81" containerName="maas-api" containerID="cri-o://a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254" gracePeriod=30 Jul 29 19:07:00.125521 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.125493 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:07:00.228595 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.228556 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-metrics-certs\") pod \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " Jul 29 19:07:00.228798 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.228635 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vvw4\" (UniqueName: \"kubernetes.io/projected/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-kube-api-access-4vvw4\") pod \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " Jul 29 19:07:00.228798 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.228664 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-openshift-service-ca\") pod \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " Jul 29 19:07:00.228798 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.228707 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-maas-api-tls\") pod \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\" (UID: \"65b9ae53-5c1c-4aad-a879-8ad8270c5a81\") " Jul 29 19:07:00.229084 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.229059 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "65b9ae53-5c1c-4aad-a879-8ad8270c5a81" (UID: "65b9ae53-5c1c-4aad-a879-8ad8270c5a81"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:07:00.230857 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.230836 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "65b9ae53-5c1c-4aad-a879-8ad8270c5a81" (UID: "65b9ae53-5c1c-4aad-a879-8ad8270c5a81"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:07:00.230963 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.230910 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "65b9ae53-5c1c-4aad-a879-8ad8270c5a81" (UID: "65b9ae53-5c1c-4aad-a879-8ad8270c5a81"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:07:00.231028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.231008 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-kube-api-access-4vvw4" (OuterVolumeSpecName: "kube-api-access-4vvw4") pod "65b9ae53-5c1c-4aad-a879-8ad8270c5a81" (UID: "65b9ae53-5c1c-4aad-a879-8ad8270c5a81"). InnerVolumeSpecName "kube-api-access-4vvw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:07:00.330335 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.330302 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:00.330335 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.330330 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:00.330335 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.330340 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vvw4\" (UniqueName: \"kubernetes.io/projected/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-kube-api-access-4vvw4\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:00.330637 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.330350 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/65b9ae53-5c1c-4aad-a879-8ad8270c5a81-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:00.823295 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.823262 2574 generic.go:356] "Generic (PLEG): container finished" podID="65b9ae53-5c1c-4aad-a879-8ad8270c5a81" containerID="a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254" exitCode=0 Jul 29 19:07:00.823502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.823314 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" event={"ID":"65b9ae53-5c1c-4aad-a879-8ad8270c5a81","Type":"ContainerDied","Data":"a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254"} Jul 29 19:07:00.823502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.823331 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" Jul 29 19:07:00.823502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.823369 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw" event={"ID":"65b9ae53-5c1c-4aad-a879-8ad8270c5a81","Type":"ContainerDied","Data":"b363f17fd82d7bf2c9a76c313bb3f85f1eca4b9c0462188851f8b15b44b742f0"} Jul 29 19:07:00.823502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.823398 2574 scope.go:117] "RemoveContainer" containerID="a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254" Jul 29 19:07:00.832608 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.832580 2574 scope.go:117] "RemoveContainer" containerID="a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254" Jul 29 19:07:00.832891 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:07:00.832872 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254\": container with ID starting with a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254 not found: ID does not exist" containerID="a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254" Jul 29 19:07:00.833030 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.832900 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254"} err="failed to get container status \"a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254\": rpc error: code = NotFound desc = could not find container \"a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254\": container with ID starting with a8cf3a35ff10080036f69786642c2a4922af89299ace8a5fec594d3a91420254 not found: ID does not exist" Jul 29 19:07:00.845402 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.845376 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw"] Jul 29 19:07:00.850318 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:00.850286 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b5792115-7c854d9bb5-q2fhw"] Jul 29 19:07:02.767730 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:02.767696 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65b9ae53-5c1c-4aad-a879-8ad8270c5a81" path="/var/lib/kubelet/pods/65b9ae53-5c1c-4aad-a879-8ad8270c5a81/volumes" Jul 29 19:07:06.874057 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:06.874025 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z"] Jul 29 19:07:06.874542 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:06.874282 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" containerID="cri-o://8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da" gracePeriod=30 Jul 29 19:07:12.129980 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.129950 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:07:12.245481 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245446 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-data\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245515 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-envoy\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245539 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-token\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245581 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e06ebf0-87ee-4013-976d-28170b8215f5-istiod-ca-cert\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245607 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-socket\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245758 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-49dwh\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-kube-api-access-49dwh\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245812 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-credential-socket\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245841 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-certs\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245879 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-podinfo\") pod \"8e06ebf0-87ee-4013-976d-28170b8215f5\" (UID: \"8e06ebf0-87ee-4013-976d-28170b8215f5\") " Jul 29 19:07:12.245958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.245908 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:07:12.246201 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246007 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e06ebf0-87ee-4013-976d-28170b8215f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:07:12.246201 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246128 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:07:12.246201 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246143 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:07:12.246420 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246352 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.246542 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246500 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.246629 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246547 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e06ebf0-87ee-4013-976d-28170b8215f5-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.246629 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246561 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.246629 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.246564 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:07:12.248111 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.248085 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:07:12.248111 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.248093 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:07:12.248313 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.248251 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-kube-api-access-49dwh" (OuterVolumeSpecName: "kube-api-access-49dwh") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "kube-api-access-49dwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:07:12.248492 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.248478 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8e06ebf0-87ee-4013-976d-28170b8215f5" (UID: "8e06ebf0-87ee-4013-976d-28170b8215f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:07:12.348067 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.348036 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.348067 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.348063 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.348067 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.348072 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.348367 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.348081 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49dwh\" (UniqueName: \"kubernetes.io/projected/8e06ebf0-87ee-4013-976d-28170b8215f5-kube-api-access-49dwh\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.348367 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.348089 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e06ebf0-87ee-4013-976d-28170b8215f5-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:07:12.883764 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.883730 2574 generic.go:356] "Generic (PLEG): container finished" podID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerID="8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da" exitCode=0 Jul 29 19:07:12.883992 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.883769 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" event={"ID":"8e06ebf0-87ee-4013-976d-28170b8215f5","Type":"ContainerDied","Data":"8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da"} Jul 29 19:07:12.883992 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.883842 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" event={"ID":"8e06ebf0-87ee-4013-976d-28170b8215f5","Type":"ContainerDied","Data":"8d35161b758844298ee3a7ba69e4fd0058c574fbba347a1865de2b23308936ea"} Jul 29 19:07:12.883992 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.883784 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z" Jul 29 19:07:12.883992 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.883873 2574 scope.go:117] "RemoveContainer" containerID="8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da" Jul 29 19:07:12.893021 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.893003 2574 scope.go:117] "RemoveContainer" containerID="58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e" Jul 29 19:07:12.901817 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.901788 2574 scope.go:117] "RemoveContainer" containerID="8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da" Jul 29 19:07:12.902183 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:07:12.902159 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da\": container with ID starting with 8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da not found: ID does not exist" containerID="8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da" Jul 29 19:07:12.902330 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.902196 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da"} err="failed to get container status \"8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da\": rpc error: code = NotFound desc = could not find container \"8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da\": container with ID starting with 8e07944078b9bf844532fc064991330d6021de623734718865a5fdad35c678da not found: ID does not exist" Jul 29 19:07:12.902330 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.902223 2574 scope.go:117] "RemoveContainer" containerID="58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e" Jul 29 19:07:12.902515 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:07:12.902496 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e\": container with ID starting with 58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e not found: ID does not exist" containerID="58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e" Jul 29 19:07:12.902555 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.902523 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e"} err="failed to get container status \"58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e\": rpc error: code = NotFound desc = could not find container \"58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e\": container with ID starting with 58e8750ca2a0d0c0bca273600937c7bfa30fa4d7b71b99011b383ee66868339e not found: ID does not exist" Jul 29 19:07:12.903270 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.903251 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z"] Jul 29 19:07:12.907962 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:12.907937 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b5792115-openshift-default-67d66f8d6d-hgv7z"] Jul 29 19:07:14.772729 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:14.772691 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" path="/var/lib/kubelet/pods/8e06ebf0-87ee-4013-976d-28170b8215f5/volumes" Jul 29 19:07:27.184992 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.184952 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh"] Jul 29 19:07:27.186068 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186037 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65b9ae53-5c1c-4aad-a879-8ad8270c5a81" containerName="maas-api" Jul 29 19:07:27.186068 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186065 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="65b9ae53-5c1c-4aad-a879-8ad8270c5a81" containerName="maas-api" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186086 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186096 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186110 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186119 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186215 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186225 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e06ebf0-87ee-4013-976d-28170b8215f5" containerName="istio-proxy" Jul 29 19:07:27.186277 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.186237 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="65b9ae53-5c1c-4aad-a879-8ad8270c5a81" containerName="maas-api" Jul 29 19:07:27.188882 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.188852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.191683 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.191661 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-8eafda81-openshift-default-dockercfg-ctptl\"" Jul 29 19:07:27.200226 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.200194 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh"] Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286552 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntsmn\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-kube-api-access-ntsmn\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286608 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-envoy\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286634 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-data\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286684 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbf5a07-7353-49a0-a343-307362b15583-istio-podinfo\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286712 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-istio-token\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286738 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-socket\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286783 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-credential-socket\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.286830 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286812 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-certs\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.287415 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.286893 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbf5a07-7353-49a0-a343-307362b15583-istiod-ca-cert\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387459 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387422 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbf5a07-7353-49a0-a343-307362b15583-istiod-ca-cert\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387486 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntsmn\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-kube-api-access-ntsmn\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387514 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-envoy\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387535 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-data\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387568 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbf5a07-7353-49a0-a343-307362b15583-istio-podinfo\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387592 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-istio-token\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-socket\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.387679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387666 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-credential-socket\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.388087 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.387697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-certs\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.388087 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.388048 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-data\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.388435 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.388159 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-certs\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.388435 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.388237 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-credential-socket\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.388435 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.388240 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-socket\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.388435 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.388282 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbf5a07-7353-49a0-a343-307362b15583-istiod-ca-cert\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.390603 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.390582 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-envoy\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.390741 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.390725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbf5a07-7353-49a0-a343-307362b15583-istio-podinfo\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.398052 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.398023 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-istio-token\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.398203 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.398142 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntsmn\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-kube-api-access-ntsmn\") pod \"e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.501187 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.501089 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:27.648942 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.648889 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh"] Jul 29 19:07:27.652189 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:07:27.652151 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbf5a07_7353_49a0_a343_307362b15583.slice/crio-b7f2631627928c04965345d45f54b62db431e3ad946fe6fbf87cad4b7e249f37 WatchSource:0}: Error finding container b7f2631627928c04965345d45f54b62db431e3ad946fe6fbf87cad4b7e249f37: Status 404 returned error can't find the container with id b7f2631627928c04965345d45f54b62db431e3ad946fe6fbf87cad4b7e249f37 Jul 29 19:07:27.654617 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.654576 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:07:27.654694 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.654669 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:07:27.654749 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.654716 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:07:27.957113 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.957077 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" event={"ID":"cbbf5a07-7353-49a0-a343-307362b15583","Type":"ContainerStarted","Data":"a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f"} Jul 29 19:07:27.957292 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.957133 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" event={"ID":"cbbf5a07-7353-49a0-a343-307362b15583","Type":"ContainerStarted","Data":"b7f2631627928c04965345d45f54b62db431e3ad946fe6fbf87cad4b7e249f37"} Jul 29 19:07:27.984401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:27.984333 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podStartSLOduration=0.984310341 podStartE2EDuration="984.310341ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:07:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:07:27.981336426 +0000 UTC m=+1987.694247954" watchObservedRunningTime="2026-07-29 19:07:27.984310341 +0000 UTC m=+1987.697221832" Jul 29 19:07:28.501665 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:28.501628 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:28.506998 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:28.506975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:28.962244 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:28.962210 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:28.963370 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:28.963349 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:30.162363 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.162328 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f"] Jul 29 19:07:30.164651 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.164635 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.167961 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.167938 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-8eafda81-serving-cert\"" Jul 29 19:07:30.177617 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.177588 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f"] Jul 29 19:07:30.214428 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.214390 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/df915bd9-d970-42f5-9175-9e3374394c7f-kube-api-access-4x7tg\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.214628 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.214446 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-metrics-certs\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.214628 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.214554 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.214628 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.214606 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df915bd9-d970-42f5-9175-9e3374394c7f-openshift-service-ca\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.315824 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.315784 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/df915bd9-d970-42f5-9175-9e3374394c7f-kube-api-access-4x7tg\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.316055 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.315832 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-metrics-certs\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.316055 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.315863 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.316055 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.315887 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df915bd9-d970-42f5-9175-9e3374394c7f-openshift-service-ca\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.316055 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:07:30.316019 2574 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-serving-cert: secret "maas-api-e2e-mt-8eafda81-serving-cert" not found Jul 29 19:07:30.316296 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:07:30.316101 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls podName:df915bd9-d970-42f5-9175-9e3374394c7f nodeName:}" failed. No retries permitted until 2026-07-29 19:07:30.816085434 +0000 UTC m=+1990.528996902 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls") pod "maas-api-e2e-mt-8eafda81-c85df74c-chl9f" (UID: "df915bd9-d970-42f5-9175-9e3374394c7f") : secret "maas-api-e2e-mt-8eafda81-serving-cert" not found Jul 29 19:07:30.316551 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.316515 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df915bd9-d970-42f5-9175-9e3374394c7f-openshift-service-ca\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.318528 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.318505 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-metrics-certs\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.328624 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.328602 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/df915bd9-d970-42f5-9175-9e3374394c7f-kube-api-access-4x7tg\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.821343 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.821307 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:30.823967 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:30.823937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls\") pod \"maas-api-e2e-mt-8eafda81-c85df74c-chl9f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:31.075768 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:31.075668 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:31.253515 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:31.253488 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f"] Jul 29 19:07:31.255863 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:07:31.255838 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf915bd9_d970_42f5_9175_9e3374394c7f.slice/crio-9e80231fba5ac4b8ebd830313f353eacde1f800be224f6ee38bb96f9b52a3e21 WatchSource:0}: Error finding container 9e80231fba5ac4b8ebd830313f353eacde1f800be224f6ee38bb96f9b52a3e21: Status 404 returned error can't find the container with id 9e80231fba5ac4b8ebd830313f353eacde1f800be224f6ee38bb96f9b52a3e21 Jul 29 19:07:31.257439 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:31.257414 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 29 19:07:31.978190 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:31.978153 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" event={"ID":"df915bd9-d970-42f5-9175-9e3374394c7f","Type":"ContainerStarted","Data":"9e80231fba5ac4b8ebd830313f353eacde1f800be224f6ee38bb96f9b52a3e21"} Jul 29 19:07:32.985420 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:32.985384 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" event={"ID":"df915bd9-d970-42f5-9175-9e3374394c7f","Type":"ContainerStarted","Data":"a8ebeab314410cd9c3783ab179a72d79f02252b5e1d2eea83ef4f3188e1231bb"} Jul 29 19:07:32.985853 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:32.985504 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:39.004002 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:39.003963 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:07:39.025263 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:39.025203 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" podStartSLOduration=7.961866225 podStartE2EDuration="9.025186568s" totalImagesPullingTime="1.063320343s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:07:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:07:33.014964828 +0000 UTC m=+1992.727876316" watchObservedRunningTime="2026-07-29 19:07:39.025186568 +0000 UTC m=+1998.738098058" Jul 29 19:07:44.590647 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:44.589948 2574 patch_prober.go:28] interesting pod/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:42306->10.133.0.52:15021: read: connection reset by peer" start-of-body= Jul 29 19:07:44.590647 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:44.590049 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": read tcp 10.133.0.2:42306->10.133.0.52:15021: read: connection reset by peer" Jul 29 19:07:45.046283 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:45.046249 2574 generic.go:356] "Generic (PLEG): container finished" podID="cbbf5a07-7353-49a0-a343-307362b15583" containerID="a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f" exitCode=0 Jul 29 19:07:45.046464 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:45.046308 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" event={"ID":"cbbf5a07-7353-49a0-a343-307362b15583","Type":"ContainerDied","Data":"a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f"} Jul 29 19:07:45.046700 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:45.046685 2574 scope.go:117] "RemoveContainer" containerID="a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f" Jul 29 19:07:45.047785 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:45.047757 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:07:45.047843 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:45.047826 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:07:45.047878 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:45.047860 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:07:46.052942 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:46.052884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" event={"ID":"cbbf5a07-7353-49a0-a343-307362b15583","Type":"ContainerStarted","Data":"fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713"} Jul 29 19:07:46.501864 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:46.501821 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:46.502253 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:46.502227 2574 patch_prober.go:28] interesting pod/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 19:07:46.502407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:46.502384 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 19:07:47.502613 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:47.502572 2574 patch_prober.go:28] interesting pod/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 19:07:47.503152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:47.503013 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 19:07:48.502452 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:48.502408 2574 patch_prober.go:28] interesting pod/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 19:07:48.502784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:48.502748 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 19:07:49.502392 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:49.502352 2574 patch_prober.go:28] interesting pod/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Jul 29 19:07:49.502714 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:49.502679 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Jul 29 19:07:50.506028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:50.505996 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:50.506464 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:50.506322 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:07:50.507107 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:07:50.507085 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:08:45.118713 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.118632 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f"] Jul 29 19:08:45.119178 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.118858 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" podUID="df915bd9-d970-42f5-9175-9e3374394c7f" containerName="maas-api" containerID="cri-o://a8ebeab314410cd9c3783ab179a72d79f02252b5e1d2eea83ef4f3188e1231bb" gracePeriod=30 Jul 29 19:08:45.344769 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.344731 2574 generic.go:356] "Generic (PLEG): container finished" podID="df915bd9-d970-42f5-9175-9e3374394c7f" containerID="a8ebeab314410cd9c3783ab179a72d79f02252b5e1d2eea83ef4f3188e1231bb" exitCode=0 Jul 29 19:08:45.344960 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.344788 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" event={"ID":"df915bd9-d970-42f5-9175-9e3374394c7f","Type":"ContainerDied","Data":"a8ebeab314410cd9c3783ab179a72d79f02252b5e1d2eea83ef4f3188e1231bb"} Jul 29 19:08:45.393826 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.393788 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:08:45.470844 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.470809 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-metrics-certs\") pod \"df915bd9-d970-42f5-9175-9e3374394c7f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " Jul 29 19:08:45.470844 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.470847 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls\") pod \"df915bd9-d970-42f5-9175-9e3374394c7f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " Jul 29 19:08:45.471115 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.470875 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/df915bd9-d970-42f5-9175-9e3374394c7f-kube-api-access-4x7tg\") pod \"df915bd9-d970-42f5-9175-9e3374394c7f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " Jul 29 19:08:45.471115 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.470997 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df915bd9-d970-42f5-9175-9e3374394c7f-openshift-service-ca\") pod \"df915bd9-d970-42f5-9175-9e3374394c7f\" (UID: \"df915bd9-d970-42f5-9175-9e3374394c7f\") " Jul 29 19:08:45.471385 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.471361 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df915bd9-d970-42f5-9175-9e3374394c7f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "df915bd9-d970-42f5-9175-9e3374394c7f" (UID: "df915bd9-d970-42f5-9175-9e3374394c7f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:08:45.473238 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.473201 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "df915bd9-d970-42f5-9175-9e3374394c7f" (UID: "df915bd9-d970-42f5-9175-9e3374394c7f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:08:45.473348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.473253 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "df915bd9-d970-42f5-9175-9e3374394c7f" (UID: "df915bd9-d970-42f5-9175-9e3374394c7f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:08:45.473348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.473289 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df915bd9-d970-42f5-9175-9e3374394c7f-kube-api-access-4x7tg" (OuterVolumeSpecName: "kube-api-access-4x7tg") pod "df915bd9-d970-42f5-9175-9e3374394c7f" (UID: "df915bd9-d970-42f5-9175-9e3374394c7f"). InnerVolumeSpecName "kube-api-access-4x7tg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:08:45.572664 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.572632 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:08:45.572664 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.572658 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df915bd9-d970-42f5-9175-9e3374394c7f-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:08:45.572664 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.572667 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x7tg\" (UniqueName: \"kubernetes.io/projected/df915bd9-d970-42f5-9175-9e3374394c7f-kube-api-access-4x7tg\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:08:45.572900 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:45.572676 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df915bd9-d970-42f5-9175-9e3374394c7f-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:08:46.351508 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:46.351478 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" Jul 29 19:08:46.351912 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:46.351505 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f" event={"ID":"df915bd9-d970-42f5-9175-9e3374394c7f","Type":"ContainerDied","Data":"9e80231fba5ac4b8ebd830313f353eacde1f800be224f6ee38bb96f9b52a3e21"} Jul 29 19:08:46.351912 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:46.351571 2574 scope.go:117] "RemoveContainer" containerID="a8ebeab314410cd9c3783ab179a72d79f02252b5e1d2eea83ef4f3188e1231bb" Jul 29 19:08:46.374890 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:46.374859 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f"] Jul 29 19:08:46.379806 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:46.379777 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-8eafda81-c85df74c-chl9f"] Jul 29 19:08:46.767823 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:08:46.767789 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df915bd9-d970-42f5-9175-9e3374394c7f" path="/var/lib/kubelet/pods/df915bd9-d970-42f5-9175-9e3374394c7f/volumes" Jul 29 19:09:36.800737 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:36.800693 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh"] Jul 29 19:09:36.801229 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:36.800984 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" containerID="cri-o://fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713" gracePeriod=30 Jul 29 19:09:42.054676 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.054652 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:09:42.105339 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105250 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-socket\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105339 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105297 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-certs\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105339 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105329 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-credential-socket\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105370 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-data\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105395 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbf5a07-7353-49a0-a343-307362b15583-istio-podinfo\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105411 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-istio-token\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105436 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-envoy\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105469 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ntsmn\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-kube-api-access-ntsmn\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105509 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbf5a07-7353-49a0-a343-307362b15583-istiod-ca-cert\") pod \"cbbf5a07-7353-49a0-a343-307362b15583\" (UID: \"cbbf5a07-7353-49a0-a343-307362b15583\") " Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105575 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:09:42.105597 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105585 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:09:42.106028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105838 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.106028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.105860 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.106174 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.106132 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:09:42.106271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.106256 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbf5a07-7353-49a0-a343-307362b15583-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:09:42.106487 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.106465 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-data" (OuterVolumeSpecName: "istio-data") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:09:42.108408 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.108371 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:09:42.108515 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.108450 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbbf5a07-7353-49a0-a343-307362b15583-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:09:42.108515 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.108457 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-istio-token" (OuterVolumeSpecName: "istio-token") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:09:42.108605 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.108561 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-kube-api-access-ntsmn" (OuterVolumeSpecName: "kube-api-access-ntsmn") pod "cbbf5a07-7353-49a0-a343-307362b15583" (UID: "cbbf5a07-7353-49a0-a343-307362b15583"). InnerVolumeSpecName "kube-api-access-ntsmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:09:42.207309 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207272 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.207309 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207303 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.207309 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207313 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbbf5a07-7353-49a0-a343-307362b15583-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.207547 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207322 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.207547 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207330 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbbf5a07-7353-49a0-a343-307362b15583-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.207547 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207338 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntsmn\" (UniqueName: \"kubernetes.io/projected/cbbf5a07-7353-49a0-a343-307362b15583-kube-api-access-ntsmn\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.207547 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.207346 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbbf5a07-7353-49a0-a343-307362b15583-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:09:42.632093 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.632061 2574 generic.go:356] "Generic (PLEG): container finished" podID="cbbf5a07-7353-49a0-a343-307362b15583" containerID="fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713" exitCode=0 Jul 29 19:09:42.632312 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.632139 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" Jul 29 19:09:42.632312 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.632151 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" event={"ID":"cbbf5a07-7353-49a0-a343-307362b15583","Type":"ContainerDied","Data":"fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713"} Jul 29 19:09:42.632312 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.632265 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh" event={"ID":"cbbf5a07-7353-49a0-a343-307362b15583","Type":"ContainerDied","Data":"b7f2631627928c04965345d45f54b62db431e3ad946fe6fbf87cad4b7e249f37"} Jul 29 19:09:42.632312 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.632305 2574 scope.go:117] "RemoveContainer" containerID="fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713" Jul 29 19:09:42.641754 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.641729 2574 scope.go:117] "RemoveContainer" containerID="a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f" Jul 29 19:09:42.650033 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.650000 2574 scope.go:117] "RemoveContainer" containerID="fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713" Jul 29 19:09:42.650287 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:09:42.650267 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713\": container with ID starting with fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713 not found: ID does not exist" containerID="fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713" Jul 29 19:09:42.650391 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.650299 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713"} err="failed to get container status \"fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713\": rpc error: code = NotFound desc = could not find container \"fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713\": container with ID starting with fc4ef3b37edd9c9b341c1af9189c1243eaa82cd603f96891b7cb4867a5acc713 not found: ID does not exist" Jul 29 19:09:42.650391 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.650325 2574 scope.go:117] "RemoveContainer" containerID="a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f" Jul 29 19:09:42.650577 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:09:42.650560 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f\": container with ID starting with a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f not found: ID does not exist" containerID="a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f" Jul 29 19:09:42.650622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.650584 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f"} err="failed to get container status \"a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f\": rpc error: code = NotFound desc = could not find container \"a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f\": container with ID starting with a9ca7b19a509bfae5b6fe921b2ed27ec164ddae9c5a2bf733132d18216f3d53f not found: ID does not exist" Jul 29 19:09:42.656741 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.656720 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh"] Jul 29 19:09:42.660519 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.660499 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-8eafda81-openshift-default-6cbc87dc64-9fvmh"] Jul 29 19:09:42.768411 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:09:42.768369 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbf5a07-7353-49a0-a343-307362b15583" path="/var/lib/kubelet/pods/cbbf5a07-7353-49a0-a343-307362b15583/volumes" Jul 29 19:10:33.513555 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513518 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw"] Jul 29 19:10:33.513972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513873 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df915bd9-d970-42f5-9175-9e3374394c7f" containerName="maas-api" Jul 29 19:10:33.513972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513884 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="df915bd9-d970-42f5-9175-9e3374394c7f" containerName="maas-api" Jul 29 19:10:33.513972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513899 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" Jul 29 19:10:33.513972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513904 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" Jul 29 19:10:33.513972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513910 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" Jul 29 19:10:33.513972 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.513944 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" Jul 29 19:10:33.514172 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.514019 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="df915bd9-d970-42f5-9175-9e3374394c7f" containerName="maas-api" Jul 29 19:10:33.514172 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.514028 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" Jul 29 19:10:33.514172 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.514033 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbf5a07-7353-49a0-a343-307362b15583" containerName="istio-proxy" Jul 29 19:10:33.516853 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.516832 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.522157 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.522132 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-e3eaaf-openshift-default-dockercfg-z4dn8\"" Jul 29 19:10:33.541548 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.541516 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw"] Jul 29 19:10:33.599475 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599402 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-credential-socket\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599475 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599479 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-socket\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599585 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-podinfo\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-data\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599630 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c5c462-38b7-4541-ae58-208acba2eba8-istiod-ca-cert\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599678 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-certs\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599723 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-token\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.599796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599755 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhrvk\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-kube-api-access-rhrvk\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.600148 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.599801 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-envoy\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701003 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.700957 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-credential-socket\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701012 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-socket\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701055 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-podinfo\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701073 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-data\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701091 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c5c462-38b7-4541-ae58-208acba2eba8-istiod-ca-cert\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701137 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-certs\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-token\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701205 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhrvk\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-kube-api-access-rhrvk\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-envoy\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701372 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-credential-socket\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701423 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-socket\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701484 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-certs\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701722 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-data\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.701834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.701828 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c5c462-38b7-4541-ae58-208acba2eba8-istiod-ca-cert\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.704116 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.704095 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-envoy\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.704405 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.704387 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-podinfo\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.718410 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.718385 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhrvk\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-kube-api-access-rhrvk\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.719273 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.719249 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-token\") pod \"e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.827037 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.826949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:33.975250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.975084 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw"] Jul 29 19:10:33.978127 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:10:33.978098 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7c5c462_38b7_4541_ae58_208acba2eba8.slice/crio-dc3358813b1018d96625801bbb40e0a6b91c82a18f438bbe9cda8ef315e716ce WatchSource:0}: Error finding container dc3358813b1018d96625801bbb40e0a6b91c82a18f438bbe9cda8ef315e716ce: Status 404 returned error can't find the container with id dc3358813b1018d96625801bbb40e0a6b91c82a18f438bbe9cda8ef315e716ce Jul 29 19:10:33.980319 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.980280 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:10:33.980388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.980351 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:10:33.980388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:33.980376 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:10:34.878168 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:34.878130 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" event={"ID":"d7c5c462-38b7-4541-ae58-208acba2eba8","Type":"ContainerStarted","Data":"1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a"} Jul 29 19:10:34.878586 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:34.878184 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" event={"ID":"d7c5c462-38b7-4541-ae58-208acba2eba8","Type":"ContainerStarted","Data":"dc3358813b1018d96625801bbb40e0a6b91c82a18f438bbe9cda8ef315e716ce"} Jul 29 19:10:34.901341 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:34.901289 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" podStartSLOduration=1.9012729080000002 podStartE2EDuration="1.901272908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:34.899593225 +0000 UTC m=+2174.612504716" watchObservedRunningTime="2026-07-29 19:10:34.901272908 +0000 UTC m=+2174.614184403" Jul 29 19:10:35.827168 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:35.827130 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:35.831858 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:35.831830 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:35.883650 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:35.883618 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:35.884743 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:35.884719 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:37.076219 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.076185 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf"] Jul 29 19:10:37.079840 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.079815 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.082552 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.082526 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-e3eaaf-serving-cert\"" Jul 29 19:10:37.092182 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.092160 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf"] Jul 29 19:10:37.234900 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.234857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.235095 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.234940 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-metrics-certs\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.235095 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.234990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.235095 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.235064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwhpd\" (UniqueName: \"kubernetes.io/projected/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-kube-api-access-xwhpd\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.336520 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.336432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.336520 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.336482 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-metrics-certs\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.336759 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.336537 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.336759 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.336571 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwhpd\" (UniqueName: \"kubernetes.io/projected/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-kube-api-access-xwhpd\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.337283 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.337260 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-openshift-service-ca\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.339194 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.339176 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-maas-api-tls\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.339250 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.339192 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-metrics-certs\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.352190 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.352169 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwhpd\" (UniqueName: \"kubernetes.io/projected/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-kube-api-access-xwhpd\") pod \"maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.391703 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.391677 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:37.539022 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.538988 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf"] Jul 29 19:10:37.542016 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:10:37.541982 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fe48164_4fc5_4ec1_b5da_9a02ccd1208a.slice/crio-2f8ff13f8f08034917e2c5bfde5b8c7d20987e60bbe047881faa568778b6870d WatchSource:0}: Error finding container 2f8ff13f8f08034917e2c5bfde5b8c7d20987e60bbe047881faa568778b6870d: Status 404 returned error can't find the container with id 2f8ff13f8f08034917e2c5bfde5b8c7d20987e60bbe047881faa568778b6870d Jul 29 19:10:37.894758 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:37.894721 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" event={"ID":"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a","Type":"ContainerStarted","Data":"2f8ff13f8f08034917e2c5bfde5b8c7d20987e60bbe047881faa568778b6870d"} Jul 29 19:10:38.901664 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:38.901626 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" event={"ID":"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a","Type":"ContainerStarted","Data":"4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37"} Jul 29 19:10:38.902088 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:38.901693 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:38.931713 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:38.931666 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" podStartSLOduration=1.550887285 podStartE2EDuration="1.931649943s" totalImagesPullingTime="380.762658ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:10:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:10:38.924566251 +0000 UTC m=+2178.637477742" watchObservedRunningTime="2026-07-29 19:10:38.931649943 +0000 UTC m=+2178.644561432" Jul 29 19:10:44.913560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:44.913520 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:10:49.959739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:49.959697 2574 generic.go:356] "Generic (PLEG): container finished" podID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerID="1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a" exitCode=0 Jul 29 19:10:49.959739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:49.959731 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" event={"ID":"d7c5c462-38b7-4541-ae58-208acba2eba8","Type":"ContainerDied","Data":"1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a"} Jul 29 19:10:49.960321 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:49.960130 2574 scope.go:117] "RemoveContainer" containerID="1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a" Jul 29 19:10:49.961092 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:49.961066 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:10:49.961159 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:49.961128 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:10:49.961196 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:49.961162 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:10:50.883725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:50.883680 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:50.966771 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:50.966735 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" event={"ID":"d7c5c462-38b7-4541-ae58-208acba2eba8","Type":"ContainerStarted","Data":"b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d"} Jul 29 19:10:51.827368 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:51.827336 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:51.828479 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:51.828455 2574 patch_prober.go:28] interesting pod/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 19:10:51.828556 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:51.828504 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 19:10:52.828047 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:52.828014 2574 patch_prober.go:28] interesting pod/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 19:10:52.828536 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:52.828074 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 19:10:53.827980 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:53.827942 2574 patch_prober.go:28] interesting pod/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 19:10:53.828161 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:53.828003 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 19:10:54.831204 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:54.831177 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:54.831626 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:54.831558 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:10:54.832174 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:10:54.832156 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:11:02.629807 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.629769 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv"] Jul 29 19:11:02.632202 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.632180 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.635315 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.635294 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5b4a8e-openshift-default-dockercfg-m4gzn\"" Jul 29 19:11:02.654662 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.654634 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv"] Jul 29 19:11:02.779100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779066 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hmdc\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-kube-api-access-4hmdc\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779100 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-podinfo\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779332 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779126 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-credential-socket\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779332 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779230 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-certs\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779332 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779269 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-socket\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779332 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779298 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e014d4d-2a46-472e-b51f-126fe160f6ea-istiod-ca-cert\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779332 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779331 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-envoy\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779534 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779366 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-data\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.779534 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.779410 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-token\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880316 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880221 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-certs\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880316 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880275 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-socket\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880316 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880306 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e014d4d-2a46-472e-b51f-126fe160f6ea-istiod-ca-cert\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880599 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-envoy\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880599 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880441 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-data\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880599 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-token\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hmdc\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-kube-api-access-4hmdc\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880637 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-podinfo\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-credential-socket\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880697 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-certs\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880753 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-socket\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.880818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.880780 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-data\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.881213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.881032 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-credential-socket\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.881213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.881084 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e014d4d-2a46-472e-b51f-126fe160f6ea-istiod-ca-cert\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.883147 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.883126 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-envoy\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.883465 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.883445 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-podinfo\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.892879 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.892851 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hmdc\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-kube-api-access-4hmdc\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.893696 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.893678 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-token\") pod \"e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:02.943063 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:02.943034 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:03.090107 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:11:03.090067 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e014d4d_2a46_472e_b51f_126fe160f6ea.slice/crio-129f95d38a42839652ddfacfe19821abee5f03c816209f6c7c8e10f6f2a3de00 WatchSource:0}: Error finding container 129f95d38a42839652ddfacfe19821abee5f03c816209f6c7c8e10f6f2a3de00: Status 404 returned error can't find the container with id 129f95d38a42839652ddfacfe19821abee5f03c816209f6c7c8e10f6f2a3de00 Jul 29 19:11:03.090226 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:03.090202 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv"] Jul 29 19:11:03.092317 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:03.092276 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:11:03.092387 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:03.092349 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:11:03.092387 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:03.092382 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:11:04.033255 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:04.033214 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" event={"ID":"9e014d4d-2a46-472e-b51f-126fe160f6ea","Type":"ContainerStarted","Data":"f4cbbc9dbb0b304a0c3a8f5ab3a20d4a099ed5609198142a9d7ba9d999ad1d76"} Jul 29 19:11:04.033255 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:04.033262 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" event={"ID":"9e014d4d-2a46-472e-b51f-126fe160f6ea","Type":"ContainerStarted","Data":"129f95d38a42839652ddfacfe19821abee5f03c816209f6c7c8e10f6f2a3de00"} Jul 29 19:11:04.056877 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:04.056823 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podStartSLOduration=2.056786065 podStartE2EDuration="2.056786065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:11:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:11:04.05516998 +0000 UTC m=+2203.768081473" watchObservedRunningTime="2026-07-29 19:11:04.056786065 +0000 UTC m=+2203.769697555" Jul 29 19:11:04.943469 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:04.943431 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:04.948804 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:04.948774 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:05.038831 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:05.038803 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:05.039696 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:05.039675 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:21.080063 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:21.080032 2574 patch_prober.go:28] interesting pod/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 19:11:21.082590 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:21.080091 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 19:11:22.138795 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:22.138758 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerID="f4cbbc9dbb0b304a0c3a8f5ab3a20d4a099ed5609198142a9d7ba9d999ad1d76" exitCode=0 Jul 29 19:11:22.139262 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:22.138831 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" event={"ID":"9e014d4d-2a46-472e-b51f-126fe160f6ea","Type":"ContainerDied","Data":"f4cbbc9dbb0b304a0c3a8f5ab3a20d4a099ed5609198142a9d7ba9d999ad1d76"} Jul 29 19:11:22.139333 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:22.139296 2574 scope.go:117] "RemoveContainer" containerID="f4cbbc9dbb0b304a0c3a8f5ab3a20d4a099ed5609198142a9d7ba9d999ad1d76" Jul 29 19:11:22.140402 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:22.140368 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:11:22.140503 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:22.140438 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:11:22.140503 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:22.140468 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:11:23.146179 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:23.146142 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" event={"ID":"9e014d4d-2a46-472e-b51f-126fe160f6ea","Type":"ContainerStarted","Data":"791bf1607f6b42996099a47e70a449ce7be680b3aaf2bf52b5af229143533b5f"} Jul 29 19:11:23.943637 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:23.943598 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:23.943868 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:23.943847 2574 patch_prober.go:28] interesting pod/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 29 19:11:23.943940 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:23.943896 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 29 19:11:24.943569 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:24.943532 2574 patch_prober.go:28] interesting pod/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 29 19:11:24.943981 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:24.943600 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 29 19:11:25.943942 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:25.943887 2574 patch_prober.go:28] interesting pod/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body= Jul 29 19:11:25.944356 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:25.943967 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" Jul 29 19:11:26.985348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:26.985316 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:26.985759 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:26.985586 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:11:26.986416 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:11:26.986396 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:12:57.685217 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:57.685092 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf"] Jul 29 19:12:57.685873 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:57.685341 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" podUID="7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" containerName="maas-api" containerID="cri-o://4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37" gracePeriod=30 Jul 29 19:12:57.952143 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:57.952115 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:12:58.030207 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.030174 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xwhpd\" (UniqueName: \"kubernetes.io/projected/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-kube-api-access-xwhpd\") pod \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " Jul 29 19:12:58.030399 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.030282 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-openshift-service-ca\") pod \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " Jul 29 19:12:58.030399 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.030328 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-maas-api-tls\") pod \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " Jul 29 19:12:58.030499 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.030412 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-metrics-certs\") pod \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\" (UID: \"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a\") " Jul 29 19:12:58.030707 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.030678 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" (UID: "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:12:58.032738 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.032710 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" (UID: "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:12:58.032738 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.032712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-kube-api-access-xwhpd" (OuterVolumeSpecName: "kube-api-access-xwhpd") pod "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" (UID: "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a"). InnerVolumeSpecName "kube-api-access-xwhpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:12:58.032738 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.032731 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" (UID: "7fe48164-4fc5-4ec1-b5da-9a02ccd1208a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 29 19:12:58.131427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.131392 2574 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-openshift-service-ca\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:12:58.131427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.131425 2574 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-maas-api-tls\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:12:58.131427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.131436 2574 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-metrics-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:12:58.131674 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.131444 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwhpd\" (UniqueName: \"kubernetes.io/projected/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a-kube-api-access-xwhpd\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:12:58.614175 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.614138 2574 generic.go:356] "Generic (PLEG): container finished" podID="7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" containerID="4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37" exitCode=0 Jul 29 19:12:58.614377 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.614193 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" Jul 29 19:12:58.614377 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.614225 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" event={"ID":"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a","Type":"ContainerDied","Data":"4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37"} Jul 29 19:12:58.614377 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.614278 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf" event={"ID":"7fe48164-4fc5-4ec1-b5da-9a02ccd1208a","Type":"ContainerDied","Data":"2f8ff13f8f08034917e2c5bfde5b8c7d20987e60bbe047881faa568778b6870d"} Jul 29 19:12:58.614377 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.614302 2574 scope.go:117] "RemoveContainer" containerID="4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37" Jul 29 19:12:58.623826 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.623808 2574 scope.go:117] "RemoveContainer" containerID="4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37" Jul 29 19:12:58.624109 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:12:58.624090 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37\": container with ID starting with 4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37 not found: ID does not exist" containerID="4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37" Jul 29 19:12:58.624183 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.624121 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37"} err="failed to get container status \"4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37\": rpc error: code = NotFound desc = could not find container \"4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37\": container with ID starting with 4f37e83df16a0d4993e9b688ed597aab068027cd0e3a387f72cc390c5604dd37 not found: ID does not exist" Jul 29 19:12:58.642743 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.642663 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf"] Jul 29 19:12:58.644102 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.644080 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-e3eaaf-7cb8c97cbb-hpzwf"] Jul 29 19:12:58.767683 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:12:58.767647 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" path="/var/lib/kubelet/pods/7fe48164-4fc5-4ec1-b5da-9a02ccd1208a/volumes" Jul 29 19:13:04.757099 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:04.757059 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw"] Jul 29 19:13:04.757509 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:04.757260 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" containerID="cri-o://b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d" gracePeriod=30 Jul 29 19:13:09.833045 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:09.833012 2574 patch_prober.go:28] interesting pod/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body= Jul 29 19:13:09.833429 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:09.833082 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" Jul 29 19:13:10.008579 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.008550 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:13:10.150502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150413 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-envoy\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.150502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150464 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-token\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.150502 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150481 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhrvk\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-kube-api-access-rhrvk\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.150755 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150523 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c5c462-38b7-4541-ae58-208acba2eba8-istiod-ca-cert\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.150755 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150541 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-credential-socket\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.150755 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150564 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-data\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.150755 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150651 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-podinfo\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.151051 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150682 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-socket\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.151407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150841 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:10.151407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.150979 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7c5c462-38b7-4541-ae58-208acba2eba8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:13:10.151407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.151060 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:10.151655 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.151628 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-certs\") pod \"d7c5c462-38b7-4541-ae58-208acba2eba8\" (UID: \"d7c5c462-38b7-4541-ae58-208acba2eba8\") " Jul 29 19:13:10.151766 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.151692 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:10.151900 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.151873 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-data" (OuterVolumeSpecName: "istio-data") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:10.152088 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152067 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d7c5c462-38b7-4541-ae58-208acba2eba8-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.152156 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152094 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.152156 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152110 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.152156 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152123 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.152156 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152136 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.153013 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152986 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-kube-api-access-rhrvk" (OuterVolumeSpecName: "kube-api-access-rhrvk") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "kube-api-access-rhrvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:10.153013 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152996 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:10.153013 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.152998 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:13:10.153226 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.153047 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-token" (OuterVolumeSpecName: "istio-token") pod "d7c5c462-38b7-4541-ae58-208acba2eba8" (UID: "d7c5c462-38b7-4541-ae58-208acba2eba8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:10.253348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.253318 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.253348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.253344 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.253348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.253352 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhrvk\" (UniqueName: \"kubernetes.io/projected/d7c5c462-38b7-4541-ae58-208acba2eba8-kube-api-access-rhrvk\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.253586 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.253361 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d7c5c462-38b7-4541-ae58-208acba2eba8-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:10.675359 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.675324 2574 generic.go:356] "Generic (PLEG): container finished" podID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerID="b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d" exitCode=0 Jul 29 19:13:10.675558 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.675387 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" Jul 29 19:13:10.675558 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.675410 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" event={"ID":"d7c5c462-38b7-4541-ae58-208acba2eba8","Type":"ContainerDied","Data":"b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d"} Jul 29 19:13:10.675558 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.675482 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw" event={"ID":"d7c5c462-38b7-4541-ae58-208acba2eba8","Type":"ContainerDied","Data":"dc3358813b1018d96625801bbb40e0a6b91c82a18f438bbe9cda8ef315e716ce"} Jul 29 19:13:10.675558 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.675509 2574 scope.go:117] "RemoveContainer" containerID="b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d" Jul 29 19:13:10.686367 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.686193 2574 scope.go:117] "RemoveContainer" containerID="1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a" Jul 29 19:13:10.696308 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.696288 2574 scope.go:117] "RemoveContainer" containerID="b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d" Jul 29 19:13:10.696560 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:13:10.696542 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d\": container with ID starting with b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d not found: ID does not exist" containerID="b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d" Jul 29 19:13:10.696628 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.696567 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d"} err="failed to get container status \"b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d\": rpc error: code = NotFound desc = could not find container \"b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d\": container with ID starting with b6cea54f6811b7d51b865cbdeabccdad973c5804a30942bd64f843cd61ca051d not found: ID does not exist" Jul 29 19:13:10.696628 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.696582 2574 scope.go:117] "RemoveContainer" containerID="1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a" Jul 29 19:13:10.696818 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:13:10.696800 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a\": container with ID starting with 1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a not found: ID does not exist" containerID="1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a" Jul 29 19:13:10.696858 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.696826 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a"} err="failed to get container status \"1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a\": rpc error: code = NotFound desc = could not find container \"1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a\": container with ID starting with 1b6282ce61603ff232e741efeff1a66ba759aa8312d65c91ccf6b18e4132cf4a not found: ID does not exist" Jul 29 19:13:10.703294 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.703272 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw"] Jul 29 19:13:10.707290 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.707268 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-e3eaaf-openshift-default-578dc94c56-bp8rw"] Jul 29 19:13:10.769699 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:10.769670 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" path="/var/lib/kubelet/pods/d7c5c462-38b7-4541-ae58-208acba2eba8/volumes" Jul 29 19:13:22.595806 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:22.595774 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv"] Jul 29 19:13:22.596247 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:22.596013 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" containerID="cri-o://791bf1607f6b42996099a47e70a449ce7be680b3aaf2bf52b5af229143533b5f" gracePeriod=30 Jul 29 19:13:23.520561 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.520529 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d"] Jul 29 19:13:23.520932 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.520904 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" Jul 29 19:13:23.521010 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521000 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" Jul 29 19:13:23.521057 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521028 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" Jul 29 19:13:23.521057 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521034 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" Jul 29 19:13:23.521057 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521040 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" containerName="maas-api" Jul 29 19:13:23.521057 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521046 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" containerName="maas-api" Jul 29 19:13:23.521176 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521106 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" Jul 29 19:13:23.521176 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521115 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d7c5c462-38b7-4541-ae58-208acba2eba8" containerName="istio-proxy" Jul 29 19:13:23.521176 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.521121 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fe48164-4fc5-4ec1-b5da-9a02ccd1208a" containerName="maas-api" Jul 29 19:13:23.524123 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.524107 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.527085 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.527065 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-1dcc27-openshift-default-dockercfg-sjtbw\"" Jul 29 19:13:23.536341 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.536315 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d"] Jul 29 19:13:23.591466 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.591423 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrjfv\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-kube-api-access-wrjfv\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.591777 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.591747 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-envoy\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.591907 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.591828 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-data\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.591907 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.591860 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-token\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.591907 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.591886 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istiod-ca-cert\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.592108 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.591942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-certs\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.592108 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.592026 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-socket\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.592108 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.592064 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-credential-socket\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.592108 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.592104 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-podinfo\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.693496 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-socket\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.693496 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693500 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-credential-socket\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-podinfo\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693566 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrjfv\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-kube-api-access-wrjfv\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-envoy\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693621 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-data\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693650 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-token\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693826 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istiod-ca-cert\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693869 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-certs\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693954 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-credential-socket\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.693988 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-socket\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694391 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.694072 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-data\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694391 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.694262 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-certs\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.694834 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.694812 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istiod-ca-cert\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.696156 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.696134 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-envoy\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.696383 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.696367 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-podinfo\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.703162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.703136 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-token\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.703427 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.703405 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrjfv\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-kube-api-access-wrjfv\") pod \"e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.835163 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.835068 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:23.977676 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.977652 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d"] Jul 29 19:13:23.979818 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:13:23.979776 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd711d0d8_caf8_48eb_b41c_dd7b380a0260.slice/crio-f31d4e79f26a10c69ee0554dd038ff91e056330cea5ac86237b11122a32d68bb WatchSource:0}: Error finding container f31d4e79f26a10c69ee0554dd038ff91e056330cea5ac86237b11122a32d68bb: Status 404 returned error can't find the container with id f31d4e79f26a10c69ee0554dd038ff91e056330cea5ac86237b11122a32d68bb Jul 29 19:13:23.982117 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.982082 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:23.982237 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.982167 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:23.982237 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:23.982215 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:24.746493 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:24.746452 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" event={"ID":"d711d0d8-caf8-48eb-b41c-dd7b380a0260","Type":"ContainerStarted","Data":"c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f"} Jul 29 19:13:24.746891 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:24.746542 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" event={"ID":"d711d0d8-caf8-48eb-b41c-dd7b380a0260","Type":"ContainerStarted","Data":"f31d4e79f26a10c69ee0554dd038ff91e056330cea5ac86237b11122a32d68bb"} Jul 29 19:13:24.767942 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:24.767873 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" podStartSLOduration=1.767858497 podStartE2EDuration="1.767858497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:24.766374717 +0000 UTC m=+2344.479286219" watchObservedRunningTime="2026-07-29 19:13:24.767858497 +0000 UTC m=+2344.480769986" Jul 29 19:13:24.836006 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:24.835975 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:25.841193 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:25.841160 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:26.758064 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:26.758028 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:26.759037 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:26.759015 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:26.988374 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:26.988338 2574 patch_prober.go:28] interesting pod/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:13:26.988958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:26.988908 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:13:27.766242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.766214 2574 generic.go:356] "Generic (PLEG): container finished" podID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerID="791bf1607f6b42996099a47e70a449ce7be680b3aaf2bf52b5af229143533b5f" exitCode=0 Jul 29 19:13:27.766402 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.766293 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" event={"ID":"9e014d4d-2a46-472e-b51f-126fe160f6ea","Type":"ContainerDied","Data":"791bf1607f6b42996099a47e70a449ce7be680b3aaf2bf52b5af229143533b5f"} Jul 29 19:13:27.766402 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.766363 2574 scope.go:117] "RemoveContainer" containerID="f4cbbc9dbb0b304a0c3a8f5ab3a20d4a099ed5609198142a9d7ba9d999ad1d76" Jul 29 19:13:27.850183 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.850160 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:13:27.937815 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.937787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-podinfo\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.937995 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.937855 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-data\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.937995 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.937972 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hmdc\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-kube-api-access-4hmdc\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938107 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.937999 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-envoy\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938107 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938055 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-token\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938107 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938088 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e014d4d-2a46-472e-b51f-126fe160f6ea-istiod-ca-cert\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938259 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938120 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-socket\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938259 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938151 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-certs\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938259 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938201 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-credential-socket\") pod \"9e014d4d-2a46-472e-b51f-126fe160f6ea\" (UID: \"9e014d4d-2a46-472e-b51f-126fe160f6ea\") " Jul 29 19:13:27.938679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938541 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:27.938977 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938708 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:27.938977 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938720 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-data" (OuterVolumeSpecName: "istio-data") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:27.938977 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938896 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:27.938977 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.938966 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e014d4d-2a46-472e-b51f-126fe160f6ea-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:13:27.940419 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.940390 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:13:27.940419 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.940390 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:13:27.940726 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.940704 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-token" (OuterVolumeSpecName: "istio-token") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:27.940808 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:27.940700 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-kube-api-access-4hmdc" (OuterVolumeSpecName: "kube-api-access-4hmdc") pod "9e014d4d-2a46-472e-b51f-126fe160f6ea" (UID: "9e014d4d-2a46-472e-b51f-126fe160f6ea"). InnerVolumeSpecName "kube-api-access-4hmdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:13:28.040066 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040031 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hmdc\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-kube-api-access-4hmdc\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040066 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040061 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040066 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040070 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040077 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e014d4d-2a46-472e-b51f-126fe160f6ea-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040086 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040094 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040102 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040110 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.040512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.040118 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e014d4d-2a46-472e-b51f-126fe160f6ea-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:13:28.775555 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.775524 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" Jul 29 19:13:28.775870 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.775554 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv" event={"ID":"9e014d4d-2a46-472e-b51f-126fe160f6ea","Type":"ContainerDied","Data":"129f95d38a42839652ddfacfe19821abee5f03c816209f6c7c8e10f6f2a3de00"} Jul 29 19:13:28.775870 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.775608 2574 scope.go:117] "RemoveContainer" containerID="791bf1607f6b42996099a47e70a449ce7be680b3aaf2bf52b5af229143533b5f" Jul 29 19:13:28.802030 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.802004 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv"] Jul 29 19:13:28.806318 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:28.806292 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5b4a8e-openshift-default-cb4869f86-khqwv"] Jul 29 19:13:30.767730 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:30.767693 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" path="/var/lib/kubelet/pods/9e014d4d-2a46-472e-b51f-126fe160f6ea/volumes" Jul 29 19:13:40.849835 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:40.849800 2574 generic.go:356] "Generic (PLEG): container finished" podID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerID="c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f" exitCode=0 Jul 29 19:13:40.850294 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:40.849884 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" event={"ID":"d711d0d8-caf8-48eb-b41c-dd7b380a0260","Type":"ContainerDied","Data":"c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f"} Jul 29 19:13:40.850393 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:40.850378 2574 scope.go:117] "RemoveContainer" containerID="c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f" Jul 29 19:13:40.851439 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:40.851408 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:40.851553 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:40.851472 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:40.851553 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:40.851505 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:41.758460 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:41.758416 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:41.856428 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:41.856395 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" event={"ID":"d711d0d8-caf8-48eb-b41c-dd7b380a0260","Type":"ContainerStarted","Data":"a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27"} Jul 29 19:13:42.836117 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:42.836083 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:42.837322 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:42.837296 2574 patch_prober.go:28] interesting pod/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 29 19:13:42.837439 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:42.837354 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 29 19:13:43.836388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:43.836346 2574 patch_prober.go:28] interesting pod/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 29 19:13:43.836881 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:43.836409 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 29 19:13:44.836129 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:44.836097 2574 patch_prober.go:28] interesting pod/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" start-of-body= Jul 29 19:13:44.836315 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:44.836150 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.57:15021/healthz/ready\": dial tcp 10.133.0.57:15021: connect: connection refused" Jul 29 19:13:45.839562 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:45.839533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:45.839974 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:45.839782 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:45.840861 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:45.840634 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:13:52.748507 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.748474 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn"] Jul 29 19:13:52.749033 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.748947 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" Jul 29 19:13:52.749033 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.748964 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" Jul 29 19:13:52.749141 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.749038 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" Jul 29 19:13:52.751258 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.751227 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.754148 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.754126 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-32da20-openshift-default-dockercfg-xf8jv\"" Jul 29 19:13:52.768483 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.768458 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn"] Jul 29 19:13:52.789556 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789521 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72fzq\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-kube-api-access-72fzq\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789562 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-certs\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789582 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-data\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-credential-socket\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789746 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-token\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789772 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-podinfo\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789787 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154655dc-b722-405b-9df8-7b9b851ae5c8-istiod-ca-cert\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.789850 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789811 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-socket\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.790073 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.789906 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-envoy\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.890982 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.890938 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72fzq\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-kube-api-access-72fzq\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.890982 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.890981 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-certs\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891222 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.890999 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-data\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891222 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891043 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-credential-socket\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891222 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891075 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-token\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891222 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891208 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-podinfo\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891463 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891252 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154655dc-b722-405b-9df8-7b9b851ae5c8-istiod-ca-cert\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891463 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891289 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-socket\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891463 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-envoy\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891463 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891395 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-certs\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891463 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891426 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-data\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891463 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891454 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-credential-socket\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891768 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891594 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-socket\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.891977 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.891949 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154655dc-b722-405b-9df8-7b9b851ae5c8-istiod-ca-cert\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.893754 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.893725 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-envoy\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.893966 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.893947 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-podinfo\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.900042 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.900019 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72fzq\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-kube-api-access-72fzq\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:52.900042 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:52.900027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-token\") pod \"e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:53.062653 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.062569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:53.212366 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.212185 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn"] Jul 29 19:13:53.214724 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:13:53.214694 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod154655dc_b722_405b_9df8_7b9b851ae5c8.slice/crio-7339b45584b8f762613ffe2502c915deb3af70d56d2903226da51cd6cd6c6576 WatchSource:0}: Error finding container 7339b45584b8f762613ffe2502c915deb3af70d56d2903226da51cd6cd6c6576: Status 404 returned error can't find the container with id 7339b45584b8f762613ffe2502c915deb3af70d56d2903226da51cd6cd6c6576 Jul 29 19:13:53.217002 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.216969 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:53.217113 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.217032 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:53.217113 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.217064 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:13:53.918003 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.917955 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" event={"ID":"154655dc-b722-405b-9df8-7b9b851ae5c8","Type":"ContainerStarted","Data":"52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5"} Jul 29 19:13:53.918388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.918025 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" event={"ID":"154655dc-b722-405b-9df8-7b9b851ae5c8","Type":"ContainerStarted","Data":"7339b45584b8f762613ffe2502c915deb3af70d56d2903226da51cd6cd6c6576"} Jul 29 19:13:53.941215 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:53.941154 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podStartSLOduration=1.941136827 podStartE2EDuration="1.941136827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:13:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:13:53.938423521 +0000 UTC m=+2373.651335012" watchObservedRunningTime="2026-07-29 19:13:53.941136827 +0000 UTC m=+2373.654048317" Jul 29 19:13:54.063113 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:54.063074 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:54.068876 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:54.068850 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:54.922862 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:54.922832 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:13:54.923945 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:13:54.923928 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:14:10.966132 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:10.966086 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 19:14:10.966540 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:10.966152 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 19:14:12.023338 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:12.023301 2574 generic.go:356] "Generic (PLEG): container finished" podID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerID="52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5" exitCode=0 Jul 29 19:14:12.023878 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:12.023372 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" event={"ID":"154655dc-b722-405b-9df8-7b9b851ae5c8","Type":"ContainerDied","Data":"52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5"} Jul 29 19:14:12.023878 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:12.023780 2574 scope.go:117] "RemoveContainer" containerID="52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5" Jul 29 19:14:12.024897 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:12.024865 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:14:12.025032 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:12.024971 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:14:12.025032 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:12.025021 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:14:13.030038 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:13.029999 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" event={"ID":"154655dc-b722-405b-9df8-7b9b851ae5c8","Type":"ContainerStarted","Data":"c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf"} Jul 29 19:14:13.063235 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:13.063202 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:14:13.063435 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:13.063413 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 19:14:13.063520 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:13.063460 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 19:14:14.063364 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:14.063323 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 19:14:14.063726 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:14.063415 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 19:14:15.063709 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:15.063670 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 19:14:15.064137 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:15.063736 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 19:14:16.063464 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:16.063430 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 19:14:16.063647 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:16.063495 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 19:14:17.063474 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:17.063434 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Jul 29 19:14:17.063857 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:17.063500 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Jul 29 19:14:18.066464 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:18.066432 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:14:18.066823 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:18.066784 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:14:18.067553 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:18.067533 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:14:39.784499 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:39.784462 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d"] Jul 29 19:14:39.785046 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:39.784718 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" containerID="cri-o://a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27" gracePeriod=30 Jul 29 19:14:45.038124 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.038092 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:14:45.191039 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.191004 2574 generic.go:356] "Generic (PLEG): container finished" podID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerID="a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27" exitCode=0 Jul 29 19:14:45.191213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.191052 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" Jul 29 19:14:45.191213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.191054 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" event={"ID":"d711d0d8-caf8-48eb-b41c-dd7b380a0260","Type":"ContainerDied","Data":"a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27"} Jul 29 19:14:45.191213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.191173 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d" event={"ID":"d711d0d8-caf8-48eb-b41c-dd7b380a0260","Type":"ContainerDied","Data":"f31d4e79f26a10c69ee0554dd038ff91e056330cea5ac86237b11122a32d68bb"} Jul 29 19:14:45.191213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.191197 2574 scope.go:117] "RemoveContainer" containerID="a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27" Jul 29 19:14:45.199713 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.199692 2574 scope.go:117] "RemoveContainer" containerID="c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f" Jul 29 19:14:45.207693 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.207676 2574 scope.go:117] "RemoveContainer" containerID="a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27" Jul 29 19:14:45.208004 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:14:45.207983 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27\": container with ID starting with a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27 not found: ID does not exist" containerID="a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27" Jul 29 19:14:45.208109 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.208013 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27"} err="failed to get container status \"a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27\": rpc error: code = NotFound desc = could not find container \"a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27\": container with ID starting with a6d913c2f5c0305e4976409d601a9f69d77512d9e86e07caa416ace99cd9ac27 not found: ID does not exist" Jul 29 19:14:45.208109 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.208033 2574 scope.go:117] "RemoveContainer" containerID="c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f" Jul 29 19:14:45.208330 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:14:45.208309 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f\": container with ID starting with c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f not found: ID does not exist" containerID="c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f" Jul 29 19:14:45.208382 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.208336 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f"} err="failed to get container status \"c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f\": rpc error: code = NotFound desc = could not find container \"c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f\": container with ID starting with c08d9751aabb9808ab1bdf767f970f2449748a322040d3fa1adfd4f8b132d18f not found: ID does not exist" Jul 29 19:14:45.213661 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213643 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-envoy\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.213760 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213674 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-token\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.213760 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213715 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-socket\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.213844 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213787 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-data\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.213844 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213816 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-podinfo\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.213940 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213851 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-certs\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.213940 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213879 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrjfv\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-kube-api-access-wrjfv\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.214047 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.213975 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istiod-ca-cert\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.214047 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.214003 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-credential-socket\") pod \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\" (UID: \"d711d0d8-caf8-48eb-b41c-dd7b380a0260\") " Jul 29 19:14:45.214150 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.214041 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:14:45.214342 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.214320 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.214407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.214328 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:14:45.214563 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.214543 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:14:45.215001 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.214975 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:14:45.215110 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.215078 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-data" (OuterVolumeSpecName: "istio-data") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:14:45.216119 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.216092 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-token" (OuterVolumeSpecName: "istio-token") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:14:45.216119 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.216113 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:14:45.216453 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.216428 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:14:45.216618 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.216598 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-kube-api-access-wrjfv" (OuterVolumeSpecName: "kube-api-access-wrjfv") pod "d711d0d8-caf8-48eb-b41c-dd7b380a0260" (UID: "d711d0d8-caf8-48eb-b41c-dd7b380a0260"). InnerVolumeSpecName "kube-api-access-wrjfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:14:45.314818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314784 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.314818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314811 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.314818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314822 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.315094 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314830 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrjfv\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-kube-api-access-wrjfv\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.315094 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314841 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.315094 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314848 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.315094 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314857 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.315094 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.314865 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d711d0d8-caf8-48eb-b41c-dd7b380a0260-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:14:45.519311 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.519278 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d"] Jul 29 19:14:45.526904 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:45.526875 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-1dcc27-openshift-default-9bb4fd484-4nj9d"] Jul 29 19:14:46.767659 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:14:46.767630 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" path="/var/lib/kubelet/pods/d711d0d8-caf8-48eb-b41c-dd7b380a0260/volumes" Jul 29 19:15:02.686791 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:02.686755 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn"] Jul 29 19:15:02.687354 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:02.686995 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" containerID="cri-o://c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf" gracePeriod=30 Jul 29 19:15:03.068877 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.068793 2574 patch_prober.go:28] interesting pod/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:15:03.069067 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.068870 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:15:03.579799 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.579767 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp"] Jul 29 19:15:03.580213 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580197 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" Jul 29 19:15:03.580305 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580217 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" Jul 29 19:15:03.580305 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580238 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" Jul 29 19:15:03.580305 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580244 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" Jul 29 19:15:03.580305 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580254 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" Jul 29 19:15:03.580305 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580259 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" Jul 29 19:15:03.580541 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580320 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e014d4d-2a46-472e-b51f-126fe160f6ea" containerName="istio-proxy" Jul 29 19:15:03.580541 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580329 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" Jul 29 19:15:03.580541 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.580338 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d711d0d8-caf8-48eb-b41c-dd7b380a0260" containerName="istio-proxy" Jul 29 19:15:03.582578 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.582559 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.585175 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.585154 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-d04442-openshift-default-dockercfg-z86pp\"" Jul 29 19:15:03.595503 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.595478 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp"] Jul 29 19:15:03.691632 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-certs\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691656 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-socket\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691724 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-data\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691826 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-istio-token\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-credential-socket\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af895778-508d-466d-b7af-8747c06211b9-istio-podinfo\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.691969 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgh6t\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-kube-api-access-mgh6t\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.692006 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-envoy\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.692133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.692059 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af895778-508d-466d-b7af-8747c06211b9-istiod-ca-cert\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793136 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793097 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgh6t\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-kube-api-access-mgh6t\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793340 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793145 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-envoy\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793340 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af895778-508d-466d-b7af-8747c06211b9-istiod-ca-cert\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793340 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793233 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-certs\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793340 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793282 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-socket\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793340 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793305 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-data\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793629 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793364 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-istio-token\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793629 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-credential-socket\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793629 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793426 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af895778-508d-466d-b7af-8747c06211b9-istio-podinfo\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793799 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793736 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-certs\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793799 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793783 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-socket\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.793903 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.793802 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-data\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.794057 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.794025 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-credential-socket\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.794261 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.794241 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af895778-508d-466d-b7af-8747c06211b9-istiod-ca-cert\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.795691 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.795670 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-envoy\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.796034 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.796018 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af895778-508d-466d-b7af-8747c06211b9-istio-podinfo\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.803195 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.803168 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgh6t\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-kube-api-access-mgh6t\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.803310 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.803196 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-istio-token\") pod \"e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:03.894598 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:03.894508 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:04.034521 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.034495 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp"] Jul 29 19:15:04.036756 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:15:04.036728 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf895778_508d_466d_b7af_8747c06211b9.slice/crio-d5946134de13c7dd479d931152025b900924c30ae7eaa132d22c7e39ca04d21e WatchSource:0}: Error finding container d5946134de13c7dd479d931152025b900924c30ae7eaa132d22c7e39ca04d21e: Status 404 returned error can't find the container with id d5946134de13c7dd479d931152025b900924c30ae7eaa132d22c7e39ca04d21e Jul 29 19:15:04.038975 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.038938 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:04.039070 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.039009 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:04.039070 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.039037 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:04.283179 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.283136 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" event={"ID":"af895778-508d-466d-b7af-8747c06211b9","Type":"ContainerStarted","Data":"47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a"} Jul 29 19:15:04.283384 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.283195 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" event={"ID":"af895778-508d-466d-b7af-8747c06211b9","Type":"ContainerStarted","Data":"d5946134de13c7dd479d931152025b900924c30ae7eaa132d22c7e39ca04d21e"} Jul 29 19:15:04.313189 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.313137 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" podStartSLOduration=1.31312137 podStartE2EDuration="1.31312137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:04.311368832 +0000 UTC m=+2444.024280337" watchObservedRunningTime="2026-07-29 19:15:04.31312137 +0000 UTC m=+2444.026032861" Jul 29 19:15:04.895199 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:04.895155 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:05.900127 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:05.900091 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:06.292388 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:06.292343 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:06.293460 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:06.293438 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:07.943560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:07.943536 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:15:08.038430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038339 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-token\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038378 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-72fzq\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-kube-api-access-72fzq\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038399 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-socket\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038447 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-data\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038473 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154655dc-b722-405b-9df8-7b9b851ae5c8-istiod-ca-cert\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038502 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-certs\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038547 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-podinfo\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038586 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-credential-socket\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.038711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038632 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-envoy\") pod \"154655dc-b722-405b-9df8-7b9b851ae5c8\" (UID: \"154655dc-b722-405b-9df8-7b9b851ae5c8\") " Jul 29 19:15:08.039068 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038750 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:15:08.039068 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.038946 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/154655dc-b722-405b-9df8-7b9b851ae5c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:15:08.039068 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.039065 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.039215 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.039085 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/154655dc-b722-405b-9df8-7b9b851ae5c8-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.039215 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.039182 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:15:08.039350 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.039230 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:15:08.039613 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.039589 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:15:08.040781 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.040758 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-kube-api-access-72fzq" (OuterVolumeSpecName: "kube-api-access-72fzq") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "kube-api-access-72fzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:15:08.040872 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.040759 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:15:08.041377 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.041353 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:15:08.041479 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.041383 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "154655dc-b722-405b-9df8-7b9b851ae5c8" (UID: "154655dc-b722-405b-9df8-7b9b851ae5c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:15:08.140157 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140122 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.140157 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140155 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.140401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140167 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72fzq\" (UniqueName: \"kubernetes.io/projected/154655dc-b722-405b-9df8-7b9b851ae5c8-kube-api-access-72fzq\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.140401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140182 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.140401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140194 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.140401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140206 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/154655dc-b722-405b-9df8-7b9b851ae5c8-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.140401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.140217 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/154655dc-b722-405b-9df8-7b9b851ae5c8-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:15:08.303711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.303617 2574 generic.go:356] "Generic (PLEG): container finished" podID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerID="c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf" exitCode=0 Jul 29 19:15:08.303711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.303678 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" Jul 29 19:15:08.303711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.303704 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" event={"ID":"154655dc-b722-405b-9df8-7b9b851ae5c8","Type":"ContainerDied","Data":"c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf"} Jul 29 19:15:08.304017 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.303756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn" event={"ID":"154655dc-b722-405b-9df8-7b9b851ae5c8","Type":"ContainerDied","Data":"7339b45584b8f762613ffe2502c915deb3af70d56d2903226da51cd6cd6c6576"} Jul 29 19:15:08.304017 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.303779 2574 scope.go:117] "RemoveContainer" containerID="c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf" Jul 29 19:15:08.314868 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.314853 2574 scope.go:117] "RemoveContainer" containerID="52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5" Jul 29 19:15:08.322694 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.322676 2574 scope.go:117] "RemoveContainer" containerID="c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf" Jul 29 19:15:08.322985 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:15:08.322959 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf\": container with ID starting with c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf not found: ID does not exist" containerID="c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf" Jul 29 19:15:08.323130 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.323000 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf"} err="failed to get container status \"c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf\": rpc error: code = NotFound desc = could not find container \"c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf\": container with ID starting with c8b3ee5de5b7462c0848ffc24b287b475097d57f91b98d70961fa4549f1569cf not found: ID does not exist" Jul 29 19:15:08.323130 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.323025 2574 scope.go:117] "RemoveContainer" containerID="52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5" Jul 29 19:15:08.323332 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:15:08.323312 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5\": container with ID starting with 52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5 not found: ID does not exist" containerID="52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5" Jul 29 19:15:08.323380 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.323338 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5"} err="failed to get container status \"52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5\": rpc error: code = NotFound desc = could not find container \"52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5\": container with ID starting with 52a49f820c7ed5fe61e8baccb5ed99f260b295aa1ab37c3c86dd3ec906d444e5 not found: ID does not exist" Jul 29 19:15:08.328818 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.328796 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn"] Jul 29 19:15:08.336256 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.336235 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-32da20-openshift-default-57c5f457f8-8xsqn"] Jul 29 19:15:08.767147 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:08.767113 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" path="/var/lib/kubelet/pods/154655dc-b722-405b-9df8-7b9b851ae5c8/volumes" Jul 29 19:15:20.366422 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:20.366388 2574 generic.go:356] "Generic (PLEG): container finished" podID="af895778-508d-466d-b7af-8747c06211b9" containerID="47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a" exitCode=0 Jul 29 19:15:20.366974 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:20.366469 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" event={"ID":"af895778-508d-466d-b7af-8747c06211b9","Type":"ContainerDied","Data":"47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a"} Jul 29 19:15:20.367081 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:20.367064 2574 scope.go:117] "RemoveContainer" containerID="47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a" Jul 29 19:15:20.368159 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:20.368125 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:20.368271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:20.368198 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:20.368271 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:20.368239 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:21.292468 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:21.292355 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:21.372599 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:21.372563 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" event={"ID":"af895778-508d-466d-b7af-8747c06211b9","Type":"ContainerStarted","Data":"848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507"} Jul 29 19:15:21.896276 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:21.896229 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:21.896503 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:21.896409 2574 patch_prober.go:28] interesting pod/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 29 19:15:21.896503 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:21.896457 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 29 19:15:22.895882 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:22.895839 2574 patch_prober.go:28] interesting pod/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 29 19:15:22.896434 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:22.895942 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 29 19:15:23.894994 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:23.894957 2574 patch_prober.go:28] interesting pod/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 29 19:15:23.895298 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:23.895268 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 29 19:15:24.895498 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:24.895462 2574 patch_prober.go:28] interesting pod/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body= Jul 29 19:15:24.896141 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:24.896105 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" Jul 29 19:15:25.899153 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:25.899124 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:25.899557 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:25.899380 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:25.900123 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:25.900101 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:15:31.977033 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.976995 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg"] Jul 29 19:15:31.977499 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.977433 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" Jul 29 19:15:31.977499 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.977451 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" Jul 29 19:15:31.977499 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.977481 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" Jul 29 19:15:31.977499 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.977486 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" Jul 29 19:15:31.977718 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.977563 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" Jul 29 19:15:31.979938 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.979897 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:31.985727 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:31.985697 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-422d30-openshift-default-dockercfg-zbzfn\"" Jul 29 19:15:32.003584 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.003551 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg"] Jul 29 19:15:32.089852 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.089820 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-socket\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.089852 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.089857 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67cc302-b603-4694-aba7-544a3c40232c-istiod-ca-cert\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090069 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.089882 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-istio-token\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090069 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.090009 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-data\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090069 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.090057 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kcnz\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-kube-api-access-4kcnz\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090176 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.090148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-certs\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090212 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.090176 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-credential-socket\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090212 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.090195 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67cc302-b603-4694-aba7-544a3c40232c-istio-podinfo\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.090282 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.090222 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-envoy\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.191334 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.191294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-certs\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.191622 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.191603 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-credential-socket\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.191727 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.191714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67cc302-b603-4694-aba7-544a3c40232c-istio-podinfo\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.191851 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.191836 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-envoy\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192013 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.191997 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-socket\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192124 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192111 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67cc302-b603-4694-aba7-544a3c40232c-istiod-ca-cert\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192212 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192200 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-istio-token\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192813 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-socket\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192813 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192286 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-credential-socket\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192813 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.191766 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-certs\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192813 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192407 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-data\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.192813 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kcnz\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-kube-api-access-4kcnz\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.193370 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192846 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-data\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.193370 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.192937 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67cc302-b603-4694-aba7-544a3c40232c-istiod-ca-cert\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.195054 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.195027 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-envoy\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.195659 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.195636 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67cc302-b603-4694-aba7-544a3c40232c-istio-podinfo\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.201404 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.201379 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-istio-token\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.201496 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.201452 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kcnz\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-kube-api-access-4kcnz\") pod \"e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.291636 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.291554 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:32.434489 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.434463 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg"] Jul 29 19:15:32.437086 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:15:32.437054 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd67cc302_b603_4694_aba7_544a3c40232c.slice/crio-6f881caaafef7889321ac1fee1062184732d3ec60427506e2822ef84bc9ef328 WatchSource:0}: Error finding container 6f881caaafef7889321ac1fee1062184732d3ec60427506e2822ef84bc9ef328: Status 404 returned error can't find the container with id 6f881caaafef7889321ac1fee1062184732d3ec60427506e2822ef84bc9ef328 Jul 29 19:15:32.439227 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.439194 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:32.439320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.439258 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:32.439320 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:32.439305 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:33.440153 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:33.440109 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" event={"ID":"d67cc302-b603-4694-aba7-544a3c40232c","Type":"ContainerStarted","Data":"c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c"} Jul 29 19:15:33.440561 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:33.440199 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" event={"ID":"d67cc302-b603-4694-aba7-544a3c40232c","Type":"ContainerStarted","Data":"6f881caaafef7889321ac1fee1062184732d3ec60427506e2822ef84bc9ef328"} Jul 29 19:15:33.462062 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:33.462009 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podStartSLOduration=2.461993718 podStartE2EDuration="2.461993718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:33.461149887 +0000 UTC m=+2473.174061403" watchObservedRunningTime="2026-07-29 19:15:33.461993718 +0000 UTC m=+2473.174905208" Jul 29 19:15:34.292096 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:34.292061 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:34.297548 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:34.297522 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:34.445728 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:34.445696 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:34.446786 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:34.446768 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:36.411690 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.411646 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn"] Jul 29 19:15:36.412287 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.412188 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="154655dc-b722-405b-9df8-7b9b851ae5c8" containerName="istio-proxy" Jul 29 19:15:36.414345 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.414325 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.417700 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.417680 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-422d30-dockercfg-q7wt9\"" Jul 29 19:15:36.417826 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.417716 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-422d30\"" Jul 29 19:15:36.431217 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.431193 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn"] Jul 29 19:15:36.538724 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.538682 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.538963 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.538945 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msd4f\" (UniqueName: \"kubernetes.io/projected/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-kube-api-access-msd4f\") pod \"payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.640360 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.640329 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.640541 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.640395 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msd4f\" (UniqueName: \"kubernetes.io/projected/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-kube-api-access-msd4f\") pod \"payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.641116 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.641093 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.649029 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.649002 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msd4f\" (UniqueName: \"kubernetes.io/projected/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-kube-api-access-msd4f\") pod \"payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.725022 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.724991 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:36.866243 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:36.866215 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn"] Jul 29 19:15:36.868946 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:15:36.868888 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4d0ac2_81ad_4d94_9edf_b92638f2444b.slice/crio-b465213a798152d2efef03caff27392f0598a3fbf9f0e1460299a4946801d6dd WatchSource:0}: Error finding container b465213a798152d2efef03caff27392f0598a3fbf9f0e1460299a4946801d6dd: Status 404 returned error can't find the container with id b465213a798152d2efef03caff27392f0598a3fbf9f0e1460299a4946801d6dd Jul 29 19:15:37.467683 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:37.467649 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" event={"ID":"dc4d0ac2-81ad-4d94-9edf-b92638f2444b","Type":"ContainerStarted","Data":"cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f"} Jul 29 19:15:37.468074 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:37.467730 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" event={"ID":"dc4d0ac2-81ad-4d94-9edf-b92638f2444b","Type":"ContainerStarted","Data":"b465213a798152d2efef03caff27392f0598a3fbf9f0e1460299a4946801d6dd"} Jul 29 19:15:37.468074 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:37.467755 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:37.486810 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:37.486752 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" podStartSLOduration=1.486734349 podStartE2EDuration="1.486734349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:15:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:15:37.486101851 +0000 UTC m=+2477.199013344" watchObservedRunningTime="2026-07-29 19:15:37.486734349 +0000 UTC m=+2477.199645839" Jul 29 19:15:48.474851 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:48.474822 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:15:53.552725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:53.552686 2574 generic.go:356] "Generic (PLEG): container finished" podID="d67cc302-b603-4694-aba7-544a3c40232c" containerID="c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c" exitCode=0 Jul 29 19:15:53.553185 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:53.552756 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" event={"ID":"d67cc302-b603-4694-aba7-544a3c40232c","Type":"ContainerDied","Data":"c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c"} Jul 29 19:15:53.553185 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:53.553140 2574 scope.go:117] "RemoveContainer" containerID="c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c" Jul 29 19:15:53.554127 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:53.554100 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:53.554200 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:53.554184 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:53.554247 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:53.554218 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:15:54.559149 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:54.559112 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" event={"ID":"d67cc302-b603-4694-aba7-544a3c40232c","Type":"ContainerStarted","Data":"ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad"} Jul 29 19:15:55.292615 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:55.292576 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:15:55.292784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:55.292724 2574 patch_prober.go:28] interesting pod/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:15:55.292784 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:55.292770 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 19:15:56.291955 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:56.291892 2574 patch_prober.go:28] interesting pod/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:15:56.292559 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:56.292528 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 19:15:57.292526 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:57.292483 2574 patch_prober.go:28] interesting pod/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:15:57.293054 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:57.292571 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 19:15:58.292630 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:58.292583 2574 patch_prober.go:28] interesting pod/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:15:58.293172 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:58.292664 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 19:15:59.292809 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:59.292769 2574 patch_prober.go:28] interesting pod/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body= Jul 29 19:15:59.293793 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:15:59.293389 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" Jul 29 19:16:00.296238 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:16:00.296199 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:16:00.296679 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:16:00.296662 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:16:00.297446 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:16:00.297423 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:25:01.581274 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.581236 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49"] Jul 29 19:25:01.584444 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.584423 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.587159 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.587137 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-cc3e80-openshift-default-dockercfg-cwp5l\"" Jul 29 19:25:01.597955 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.597912 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49"] Jul 29 19:25:01.743979 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.743946 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-envoy\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744157 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744025 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qpss\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-kube-api-access-9qpss\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744157 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-podinfo\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744157 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-token\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744163 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-data\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744207 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-socket\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744269 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istiod-ca-cert\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744311 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-certs\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.744514 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.744365 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-credential-socket\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845667 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845570 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-certs\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845667 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-credential-socket\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845677 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-envoy\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845751 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qpss\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-kube-api-access-9qpss\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845782 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-podinfo\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845831 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-token\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845860 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-data\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.845905 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845893 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-socket\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.846242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.845978 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istiod-ca-cert\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.846242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.846081 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-certs\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.846242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.846129 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-credential-socket\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.846393 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.846264 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-data\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.846393 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.846342 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-socket\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.846565 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.846548 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istiod-ca-cert\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.848379 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.848361 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-envoy\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.848532 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.848514 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-podinfo\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.855798 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.855778 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qpss\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-kube-api-access-9qpss\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.855906 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.855811 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-token\") pod \"e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:01.897986 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:01.897962 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:02.041325 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.041301 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49"] Jul 29 19:25:02.043581 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:25:02.043547 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-9fa966941a7531ad059f9fa2ec3b904505b73540542058027369488924bbe335 WatchSource:0}: Error finding container 9fa966941a7531ad059f9fa2ec3b904505b73540542058027369488924bbe335: Status 404 returned error can't find the container with id 9fa966941a7531ad059f9fa2ec3b904505b73540542058027369488924bbe335 Jul 29 19:25:02.045987 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.045903 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:02.046077 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.046032 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:02.046077 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.046062 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:02.251351 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.251311 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" event={"ID":"e5df3d34-8c16-4655-a8f5-1e3ffc25c578","Type":"ContainerStarted","Data":"0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f"} Jul 29 19:25:02.251542 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.251368 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" event={"ID":"e5df3d34-8c16-4655-a8f5-1e3ffc25c578","Type":"ContainerStarted","Data":"9fa966941a7531ad059f9fa2ec3b904505b73540542058027369488924bbe335"} Jul 29 19:25:02.282106 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.282043 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podStartSLOduration=1.282015628 podStartE2EDuration="1.282015628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:02.278902359 +0000 UTC m=+3041.991813851" watchObservedRunningTime="2026-07-29 19:25:02.282015628 +0000 UTC m=+3041.994927120" Jul 29 19:25:02.898795 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:02.898761 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:03.904572 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:03.904542 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:04.265024 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:04.264990 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:04.265902 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:04.265881 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:20.306118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:20.306066 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 29 19:25:20.306711 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:20.306153 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 29 19:25:21.351587 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:21.351548 2574 generic.go:356] "Generic (PLEG): container finished" podID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerID="0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f" exitCode=0 Jul 29 19:25:21.351587 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:21.351592 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" event={"ID":"e5df3d34-8c16-4655-a8f5-1e3ffc25c578","Type":"ContainerDied","Data":"0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f"} Jul 29 19:25:21.352133 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:21.352006 2574 scope.go:117] "RemoveContainer" containerID="0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f" Jul 29 19:25:21.353178 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:21.353146 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:21.353242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:21.353228 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:21.353279 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:21.353263 2574 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892160Ki","pods":"250"} Jul 29 19:25:22.359200 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:22.359161 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" event={"ID":"e5df3d34-8c16-4655-a8f5-1e3ffc25c578","Type":"ContainerStarted","Data":"4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db"} Jul 29 19:25:22.899404 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:22.899363 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:22.899633 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:22.899522 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:22.899633 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:22.899580 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 19:25:23.898981 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:23.898935 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:23.899483 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:23.899005 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 19:25:24.899170 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:24.899131 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:24.899789 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:24.899754 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 19:25:25.635982 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:25.635942 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:25.898657 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:25.898552 2574 patch_prober.go:28] interesting pod/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body= Jul 29 19:25:25.898939 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:25.898886 2574 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" Jul 29 19:25:26.902588 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:26.902557 2574 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:26.903071 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:26.902947 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:26.903514 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:26.903494 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:25:28.693422 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:28.693386 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:37.575793 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.575754 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf"] Jul 29 19:25:37.578727 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.578703 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.582511 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.582481 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f0bee1-dockercfg-nqrhz\"" Jul 29 19:25:37.582511 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.582480 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f0bee1\"" Jul 29 19:25:37.591201 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.591163 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf"] Jul 29 19:25:37.706812 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.706761 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sr6z4\" (UniqueName: \"kubernetes.io/projected/272b63a2-c286-45a8-885e-076c852f9398-kube-api-access-sr6z4\") pod \"payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.707041 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.706835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b63a2-c286-45a8-885e-076c852f9398-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.808449 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.808405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sr6z4\" (UniqueName: \"kubernetes.io/projected/272b63a2-c286-45a8-885e-076c852f9398-kube-api-access-sr6z4\") pod \"payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.808643 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.808456 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b63a2-c286-45a8-885e-076c852f9398-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.809267 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.809245 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b63a2-c286-45a8-885e-076c852f9398-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.817444 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.817418 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sr6z4\" (UniqueName: \"kubernetes.io/projected/272b63a2-c286-45a8-885e-076c852f9398-kube-api-access-sr6z4\") pod \"payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:37.892118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:37.892013 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:38.036682 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:38.036655 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf"] Jul 29 19:25:38.038726 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:25:38.038696 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272b63a2_c286_45a8_885e_076c852f9398.slice/crio-fb803b85a4ac9d66735c72f6c9383ab1109661036bd08c6c6cc89b32c1359d12 WatchSource:0}: Error finding container fb803b85a4ac9d66735c72f6c9383ab1109661036bd08c6c6cc89b32c1359d12: Status 404 returned error can't find the container with id fb803b85a4ac9d66735c72f6c9383ab1109661036bd08c6c6cc89b32c1359d12 Jul 29 19:25:38.439360 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:38.439316 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" event={"ID":"272b63a2-c286-45a8-885e-076c852f9398","Type":"ContainerStarted","Data":"2f9aa2eba1a394e3f4c39938f8462f8e3c77875b3f05ea925ff2393817292841"} Jul 29 19:25:38.439568 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:38.439388 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:38.439568 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:38.439415 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" event={"ID":"272b63a2-c286-45a8-885e-076c852f9398","Type":"ContainerStarted","Data":"fb803b85a4ac9d66735c72f6c9383ab1109661036bd08c6c6cc89b32c1359d12"} Jul 29 19:25:38.457989 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:38.457939 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" podStartSLOduration=1.457897873 podStartE2EDuration="1.457897873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:25:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:25:38.456484886 +0000 UTC m=+3078.169396393" watchObservedRunningTime="2026-07-29 19:25:38.457897873 +0000 UTC m=+3078.170809363" Jul 29 19:25:38.743766 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:38.743673 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:40.586945 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:40.586894 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:45.803583 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:45.803542 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:45.803972 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:45.803694 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:48.753352 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:48.753317 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:49.446543 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:25:49.446514 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:25:55.623461 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:55.623427 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:25:58.763000 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:25:58.762963 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:26:08.810893 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:26:08.810858 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:26:08.990620 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:26:08.990592 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-bps2k_8b2eeae5-b163-4f47-9581-096c74aaafca/payload-processing/0.log" Jul 29 19:26:09.111770 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:26:09.111693 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf_272b63a2-c286-45a8-885e-076c852f9398/payload-processing/0.log" Jul 29 19:26:10.579392 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:26:10.579359 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:26:18.819804 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:26:18.819768 2574 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5df3d34_8c16_4655_a8f5_1e3ffc25c578.slice/crio-conmon-0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f.scope\": RecentStats: unable to find data in memory cache]" Jul 29 19:26:59.331435 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:26:59.331314 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-bps2k_8b2eeae5-b163-4f47-9581-096c74aaafca/payload-processing/0.log" Jul 29 19:26:59.458723 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:26:59.458694 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf_272b63a2-c286-45a8-885e-076c852f9398/payload-processing/0.log" Jul 29 19:27:04.076581 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.076540 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf"] Jul 29 19:27:04.080235 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.080202 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.083419 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.083394 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-21dbec\"" Jul 29 19:27:04.094043 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.094020 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf"] Jul 29 19:27:04.102885 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.102848 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz"] Jul 29 19:27:04.107194 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.107175 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.109997 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.109966 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-21dbec-dockercfg-4q2js\"" Jul 29 19:27:04.121361 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.121333 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz"] Jul 29 19:27:04.147512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.147484 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlgth\" (UniqueName: \"kubernetes.io/projected/f733a38f-98f0-4f57-b01f-816a953fa485-kube-api-access-tlgth\") pod \"payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.147646 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.147523 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42357943-4a3a-4476-8c16-a0d9e28c6a47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.147646 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.147618 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2vqd\" (UniqueName: \"kubernetes.io/projected/42357943-4a3a-4476-8c16-a0d9e28c6a47-kube-api-access-w2vqd\") pod \"payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.147732 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.147662 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f733a38f-98f0-4f57-b01f-816a953fa485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.248676 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.248645 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42357943-4a3a-4476-8c16-a0d9e28c6a47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.248874 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.248753 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2vqd\" (UniqueName: \"kubernetes.io/projected/42357943-4a3a-4476-8c16-a0d9e28c6a47-kube-api-access-w2vqd\") pod \"payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.248874 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.248819 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f733a38f-98f0-4f57-b01f-816a953fa485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.249036 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.248875 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlgth\" (UniqueName: \"kubernetes.io/projected/f733a38f-98f0-4f57-b01f-816a953fa485-kube-api-access-tlgth\") pod \"payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.249455 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.249434 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42357943-4a3a-4476-8c16-a0d9e28c6a47-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.249520 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.249503 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f733a38f-98f0-4f57-b01f-816a953fa485-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.261599 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.261573 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2vqd\" (UniqueName: \"kubernetes.io/projected/42357943-4a3a-4476-8c16-a0d9e28c6a47-kube-api-access-w2vqd\") pod \"payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.262357 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.262336 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlgth\" (UniqueName: \"kubernetes.io/projected/f733a38f-98f0-4f57-b01f-816a953fa485-kube-api-access-tlgth\") pod \"payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.392549 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.392469 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.418266 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.418241 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.555073 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.555035 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf"] Jul 29 19:27:04.556452 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:27:04.556424 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf733a38f_98f0_4f57_b01f_816a953fa485.slice/crio-d38606ffe88dbc4be67ddc392cb08fec0ef18a0af5afc5a8b4f15f82e8e31035 WatchSource:0}: Error finding container d38606ffe88dbc4be67ddc392cb08fec0ef18a0af5afc5a8b4f15f82e8e31035: Status 404 returned error can't find the container with id d38606ffe88dbc4be67ddc392cb08fec0ef18a0af5afc5a8b4f15f82e8e31035 Jul 29 19:27:04.785615 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.785591 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz"] Jul 29 19:27:04.788504 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:27:04.788473 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42357943_4a3a_4476_8c16_a0d9e28c6a47.slice/crio-af0d0756b5bf4f2d83bcd31ed8dbd5b76539958a0de4e59a0268f46eb34794cb WatchSource:0}: Error finding container af0d0756b5bf4f2d83bcd31ed8dbd5b76539958a0de4e59a0268f46eb34794cb: Status 404 returned error can't find the container with id af0d0756b5bf4f2d83bcd31ed8dbd5b76539958a0de4e59a0268f46eb34794cb Jul 29 19:27:04.857145 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.857118 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" event={"ID":"42357943-4a3a-4476-8c16-a0d9e28c6a47","Type":"ContainerStarted","Data":"c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5"} Jul 29 19:27:04.857269 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.857168 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" event={"ID":"42357943-4a3a-4476-8c16-a0d9e28c6a47","Type":"ContainerStarted","Data":"af0d0756b5bf4f2d83bcd31ed8dbd5b76539958a0de4e59a0268f46eb34794cb"} Jul 29 19:27:04.857269 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.857216 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:04.858723 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.858701 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" event={"ID":"f733a38f-98f0-4f57-b01f-816a953fa485","Type":"ContainerStarted","Data":"8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157"} Jul 29 19:27:04.858836 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.858743 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" event={"ID":"f733a38f-98f0-4f57-b01f-816a953fa485","Type":"ContainerStarted","Data":"d38606ffe88dbc4be67ddc392cb08fec0ef18a0af5afc5a8b4f15f82e8e31035"} Jul 29 19:27:04.858836 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.858774 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:04.875409 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.875351 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" podStartSLOduration=0.875332105 podStartE2EDuration="875.332105ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:04.873896173 +0000 UTC m=+3164.586807675" watchObservedRunningTime="2026-07-29 19:27:04.875332105 +0000 UTC m=+3164.588243596" Jul 29 19:27:04.894653 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:04.894600 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" podStartSLOduration=0.894584005 podStartE2EDuration="894.584005ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-29 19:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:27:04.892220823 +0000 UTC m=+3164.605132314" watchObservedRunningTime="2026-07-29 19:27:04.894584005 +0000 UTC m=+3164.607495537" Jul 29 19:27:15.866266 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:15.866232 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:15.866644 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:15.866436 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:34.282731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.282691 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz"] Jul 29 19:27:34.283343 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.282965 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" podUID="42357943-4a3a-4476-8c16-a0d9e28c6a47" containerName="payload-processing" containerID="cri-o://c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5" gracePeriod=30 Jul 29 19:27:34.289805 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.289771 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf"] Jul 29 19:27:34.290053 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.290024 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" podUID="f733a38f-98f0-4f57-b01f-816a953fa485" containerName="payload-pre-processing" containerID="cri-o://8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157" gracePeriod=30 Jul 29 19:27:34.580306 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.580279 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:34.616864 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.616828 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:34.647130 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647096 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f733a38f-98f0-4f57-b01f-816a953fa485-plugins-config-volume\") pod \"f733a38f-98f0-4f57-b01f-816a953fa485\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " Jul 29 19:27:34.647304 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647165 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42357943-4a3a-4476-8c16-a0d9e28c6a47-plugins-config-volume\") pod \"42357943-4a3a-4476-8c16-a0d9e28c6a47\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " Jul 29 19:27:34.647357 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647305 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w2vqd\" (UniqueName: \"kubernetes.io/projected/42357943-4a3a-4476-8c16-a0d9e28c6a47-kube-api-access-w2vqd\") pod \"42357943-4a3a-4476-8c16-a0d9e28c6a47\" (UID: \"42357943-4a3a-4476-8c16-a0d9e28c6a47\") " Jul 29 19:27:34.647357 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647335 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlgth\" (UniqueName: \"kubernetes.io/projected/f733a38f-98f0-4f57-b01f-816a953fa485-kube-api-access-tlgth\") pod \"f733a38f-98f0-4f57-b01f-816a953fa485\" (UID: \"f733a38f-98f0-4f57-b01f-816a953fa485\") " Jul 29 19:27:34.647589 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647562 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f733a38f-98f0-4f57-b01f-816a953fa485-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f733a38f-98f0-4f57-b01f-816a953fa485" (UID: "f733a38f-98f0-4f57-b01f-816a953fa485"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:27:34.647589 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647569 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42357943-4a3a-4476-8c16-a0d9e28c6a47-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "42357943-4a3a-4476-8c16-a0d9e28c6a47" (UID: "42357943-4a3a-4476-8c16-a0d9e28c6a47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:27:34.647725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647708 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f733a38f-98f0-4f57-b01f-816a953fa485-plugins-config-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:27:34.647759 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.647729 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42357943-4a3a-4476-8c16-a0d9e28c6a47-plugins-config-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:27:34.649738 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.649703 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f733a38f-98f0-4f57-b01f-816a953fa485-kube-api-access-tlgth" (OuterVolumeSpecName: "kube-api-access-tlgth") pod "f733a38f-98f0-4f57-b01f-816a953fa485" (UID: "f733a38f-98f0-4f57-b01f-816a953fa485"). InnerVolumeSpecName "kube-api-access-tlgth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:27:34.649738 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.649719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42357943-4a3a-4476-8c16-a0d9e28c6a47-kube-api-access-w2vqd" (OuterVolumeSpecName: "kube-api-access-w2vqd") pod "42357943-4a3a-4476-8c16-a0d9e28c6a47" (UID: "42357943-4a3a-4476-8c16-a0d9e28c6a47"). InnerVolumeSpecName "kube-api-access-w2vqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:27:34.749075 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.749040 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2vqd\" (UniqueName: \"kubernetes.io/projected/42357943-4a3a-4476-8c16-a0d9e28c6a47-kube-api-access-w2vqd\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:27:34.749075 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:34.749070 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlgth\" (UniqueName: \"kubernetes.io/projected/f733a38f-98f0-4f57-b01f-816a953fa485-kube-api-access-tlgth\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:27:35.024710 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.024676 2574 generic.go:356] "Generic (PLEG): container finished" podID="f733a38f-98f0-4f57-b01f-816a953fa485" containerID="8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157" exitCode=0 Jul 29 19:27:35.024892 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.024719 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" Jul 29 19:27:35.024892 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.024762 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" event={"ID":"f733a38f-98f0-4f57-b01f-816a953fa485","Type":"ContainerDied","Data":"8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157"} Jul 29 19:27:35.024892 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.024816 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf" event={"ID":"f733a38f-98f0-4f57-b01f-816a953fa485","Type":"ContainerDied","Data":"d38606ffe88dbc4be67ddc392cb08fec0ef18a0af5afc5a8b4f15f82e8e31035"} Jul 29 19:27:35.024892 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.024839 2574 scope.go:117] "RemoveContainer" containerID="8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157" Jul 29 19:27:35.026132 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.026108 2574 generic.go:356] "Generic (PLEG): container finished" podID="42357943-4a3a-4476-8c16-a0d9e28c6a47" containerID="c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5" exitCode=0 Jul 29 19:27:35.026266 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.026144 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" event={"ID":"42357943-4a3a-4476-8c16-a0d9e28c6a47","Type":"ContainerDied","Data":"c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5"} Jul 29 19:27:35.026266 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.026162 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" Jul 29 19:27:35.026266 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.026196 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz" event={"ID":"42357943-4a3a-4476-8c16-a0d9e28c6a47","Type":"ContainerDied","Data":"af0d0756b5bf4f2d83bcd31ed8dbd5b76539958a0de4e59a0268f46eb34794cb"} Jul 29 19:27:35.033938 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.033902 2574 scope.go:117] "RemoveContainer" containerID="8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157" Jul 29 19:27:35.034184 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:27:35.034152 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157\": container with ID starting with 8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157 not found: ID does not exist" containerID="8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157" Jul 29 19:27:35.034284 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.034183 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157"} err="failed to get container status \"8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157\": rpc error: code = NotFound desc = could not find container \"8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157\": container with ID starting with 8109c9343d6894277a561d5d085aad3f1e1f63201a800d4f27f6f1b8e4640157 not found: ID does not exist" Jul 29 19:27:35.034284 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.034201 2574 scope.go:117] "RemoveContainer" containerID="c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5" Jul 29 19:27:35.042994 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.042974 2574 scope.go:117] "RemoveContainer" containerID="c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5" Jul 29 19:27:35.043250 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:27:35.043234 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5\": container with ID starting with c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5 not found: ID does not exist" containerID="c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5" Jul 29 19:27:35.043327 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.043257 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5"} err="failed to get container status \"c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5\": rpc error: code = NotFound desc = could not find container \"c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5\": container with ID starting with c4ce565b6196a628992655758a0bfd499c1c5bc26b4054c81a54f5757a4bc8f5 not found: ID does not exist" Jul 29 19:27:35.044110 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.044089 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf"] Jul 29 19:27:35.047702 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.047682 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-21dbec-576989c85b-nbklf"] Jul 29 19:27:35.058988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.058963 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz"] Jul 29 19:27:35.062791 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:35.062768 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-21dbec-5d7d9456b-sxhvz"] Jul 29 19:27:36.767902 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:36.767865 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42357943-4a3a-4476-8c16-a0d9e28c6a47" path="/var/lib/kubelet/pods/42357943-4a3a-4476-8c16-a0d9e28c6a47/volumes" Jul 29 19:27:36.768377 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:27:36.768269 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f733a38f-98f0-4f57-b01f-816a953fa485" path="/var/lib/kubelet/pods/f733a38f-98f0-4f57-b01f-816a953fa485/volumes" Jul 29 19:28:49.499937 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:49.499889 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49"] Jul 29 19:28:49.500466 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:49.500156 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" containerID="cri-o://4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db" gracePeriod=30 Jul 29 19:28:54.754471 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.754443 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:28:54.845815 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845736 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-podinfo\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.845815 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845776 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istiod-ca-cert\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.845815 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845804 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-envoy\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845842 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-data\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845874 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-credential-socket\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845941 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-certs\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.845994 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-socket\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846024 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qpss\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-kube-api-access-9qpss\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846118 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846070 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-token\") pod \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\" (UID: \"e5df3d34-8c16-4655-a8f5-1e3ffc25c578\") " Jul 29 19:28:54.846430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846187 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:54.846430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846232 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:54.846430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846292 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:54.846430 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846320 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:28:54.846640 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846534 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.846640 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846553 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.846640 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846566 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.846640 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.846579 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.847178 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.847150 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-data" (OuterVolumeSpecName: "istio-data") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:54.848438 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.848418 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:28:54.848438 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.848426 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-token" (OuterVolumeSpecName: "istio-token") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:28:54.848671 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.848655 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-kube-api-access-9qpss" (OuterVolumeSpecName: "kube-api-access-9qpss") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "kube-api-access-9qpss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:28:54.848725 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.848654 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e5df3d34-8c16-4655-a8f5-1e3ffc25c578" (UID: "e5df3d34-8c16-4655-a8f5-1e3ffc25c578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:28:54.947884 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.947849 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.947884 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.947876 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.947884 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.947886 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qpss\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-kube-api-access-9qpss\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.947884 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.947895 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:54.948180 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:54.947904 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5df3d34-8c16-4655-a8f5-1e3ffc25c578-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:28:55.437345 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.437308 2574 generic.go:356] "Generic (PLEG): container finished" podID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerID="4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db" exitCode=0 Jul 29 19:28:55.437528 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.437371 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" Jul 29 19:28:55.437528 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.437400 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" event={"ID":"e5df3d34-8c16-4655-a8f5-1e3ffc25c578","Type":"ContainerDied","Data":"4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db"} Jul 29 19:28:55.437528 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.437473 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49" event={"ID":"e5df3d34-8c16-4655-a8f5-1e3ffc25c578","Type":"ContainerDied","Data":"9fa966941a7531ad059f9fa2ec3b904505b73540542058027369488924bbe335"} Jul 29 19:28:55.437528 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.437496 2574 scope.go:117] "RemoveContainer" containerID="4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db" Jul 29 19:28:55.447178 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.447164 2574 scope.go:117] "RemoveContainer" containerID="0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f" Jul 29 19:28:55.454898 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.454884 2574 scope.go:117] "RemoveContainer" containerID="4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db" Jul 29 19:28:55.455169 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:28:55.455150 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db\": container with ID starting with 4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db not found: ID does not exist" containerID="4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db" Jul 29 19:28:55.455242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.455177 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db"} err="failed to get container status \"4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db\": rpc error: code = NotFound desc = could not find container \"4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db\": container with ID starting with 4db4afd9462e4a8d88db81ac16f6ec0ac92df318421227e3b3454a5851ec11db not found: ID does not exist" Jul 29 19:28:55.455242 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.455195 2574 scope.go:117] "RemoveContainer" containerID="0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f" Jul 29 19:28:55.455426 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:28:55.455409 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f\": container with ID starting with 0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f not found: ID does not exist" containerID="0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f" Jul 29 19:28:55.455465 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.455433 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f"} err="failed to get container status \"0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f\": rpc error: code = NotFound desc = could not find container \"0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f\": container with ID starting with 0cf09bf4396cde6e8c951a5c6757dad1a4dbad760d04433484c2adc7d4aa588f not found: ID does not exist" Jul 29 19:28:55.460152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.460133 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49"] Jul 29 19:28:55.464214 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:55.464191 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-cc3e80-openshift-default-557987db46-jtp49"] Jul 29 19:28:56.767350 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:28:56.767316 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" path="/var/lib/kubelet/pods/e5df3d34-8c16-4655-a8f5-1e3ffc25c578/volumes" Jul 29 19:29:01.306514 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.306477 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf"] Jul 29 19:29:01.307044 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.306745 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" podUID="272b63a2-c286-45a8-885e-076c852f9398" containerName="payload-processing" containerID="cri-o://2f9aa2eba1a394e3f4c39938f8462f8e3c77875b3f05ea925ff2393817292841" gracePeriod=30 Jul 29 19:29:01.469177 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.469138 2574 generic.go:356] "Generic (PLEG): container finished" podID="272b63a2-c286-45a8-885e-076c852f9398" containerID="2f9aa2eba1a394e3f4c39938f8462f8e3c77875b3f05ea925ff2393817292841" exitCode=0 Jul 29 19:29:01.469363 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.469214 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" event={"ID":"272b63a2-c286-45a8-885e-076c852f9398","Type":"ContainerDied","Data":"2f9aa2eba1a394e3f4c39938f8462f8e3c77875b3f05ea925ff2393817292841"} Jul 29 19:29:01.572993 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.572967 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:29:01.716823 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.716789 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sr6z4\" (UniqueName: \"kubernetes.io/projected/272b63a2-c286-45a8-885e-076c852f9398-kube-api-access-sr6z4\") pod \"272b63a2-c286-45a8-885e-076c852f9398\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " Jul 29 19:29:01.717049 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.716882 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b63a2-c286-45a8-885e-076c852f9398-plugins-config-volume\") pod \"272b63a2-c286-45a8-885e-076c852f9398\" (UID: \"272b63a2-c286-45a8-885e-076c852f9398\") " Jul 29 19:29:01.717279 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.717253 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272b63a2-c286-45a8-885e-076c852f9398-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "272b63a2-c286-45a8-885e-076c852f9398" (UID: "272b63a2-c286-45a8-885e-076c852f9398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:01.719183 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.719156 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/272b63a2-c286-45a8-885e-076c852f9398-kube-api-access-sr6z4" (OuterVolumeSpecName: "kube-api-access-sr6z4") pod "272b63a2-c286-45a8-885e-076c852f9398" (UID: "272b63a2-c286-45a8-885e-076c852f9398"). InnerVolumeSpecName "kube-api-access-sr6z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:01.818345 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.818250 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/272b63a2-c286-45a8-885e-076c852f9398-plugins-config-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:01.818345 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:01.818287 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sr6z4\" (UniqueName: \"kubernetes.io/projected/272b63a2-c286-45a8-885e-076c852f9398-kube-api-access-sr6z4\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:02.474962 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:02.474913 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" Jul 29 19:29:02.474962 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:02.474958 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf" event={"ID":"272b63a2-c286-45a8-885e-076c852f9398","Type":"ContainerDied","Data":"fb803b85a4ac9d66735c72f6c9383ab1109661036bd08c6c6cc89b32c1359d12"} Jul 29 19:29:02.475465 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:02.475020 2574 scope.go:117] "RemoveContainer" containerID="2f9aa2eba1a394e3f4c39938f8462f8e3c77875b3f05ea925ff2393817292841" Jul 29 19:29:02.499371 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:02.499336 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf"] Jul 29 19:29:02.504437 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:02.504416 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f0bee1-578bf8d8cd-vkzcf"] Jul 29 19:29:02.768539 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:02.768458 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272b63a2-c286-45a8-885e-076c852f9398" path="/var/lib/kubelet/pods/272b63a2-c286-45a8-885e-076c852f9398/volumes" Jul 29 19:29:25.995336 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:25.995297 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp"] Jul 29 19:29:25.995893 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:25.995532 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" containerID="cri-o://848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507" gracePeriod=30 Jul 29 19:29:31.254399 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.254371 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:29:31.316086 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316004 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-credential-socket\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316086 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316043 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-envoy\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316086 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316077 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af895778-508d-466d-b7af-8747c06211b9-istio-podinfo\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316101 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-socket\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316166 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgh6t\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-kube-api-access-mgh6t\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316210 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-data\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316234 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-istio-token\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316265 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af895778-508d-466d-b7af-8747c06211b9-istiod-ca-cert\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316291 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:31.316358 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316309 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-certs\") pod \"af895778-508d-466d-b7af-8747c06211b9\" (UID: \"af895778-508d-466d-b7af-8747c06211b9\") " Jul 29 19:29:31.316739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316425 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:31.316739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316622 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:31.316739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316713 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.316739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316712 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af895778-508d-466d-b7af-8747c06211b9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:31.316739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316731 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.317224 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.316747 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.317478 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.317449 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-data" (OuterVolumeSpecName: "istio-data") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:31.318419 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.318398 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af895778-508d-466d-b7af-8747c06211b9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:29:31.318546 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.318522 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:31.318590 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.318544 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-kube-api-access-mgh6t" (OuterVolumeSpecName: "kube-api-access-mgh6t") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "kube-api-access-mgh6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:31.318731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.318719 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-istio-token" (OuterVolumeSpecName: "istio-token") pod "af895778-508d-466d-b7af-8747c06211b9" (UID: "af895778-508d-466d-b7af-8747c06211b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:31.417285 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.417252 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.417285 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.417280 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af895778-508d-466d-b7af-8747c06211b9-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.417285 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.417289 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgh6t\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-kube-api-access-mgh6t\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.417512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.417300 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af895778-508d-466d-b7af-8747c06211b9-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.417512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.417308 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af895778-508d-466d-b7af-8747c06211b9-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.417512 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.417316 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af895778-508d-466d-b7af-8747c06211b9-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:31.626544 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.626460 2574 generic.go:356] "Generic (PLEG): container finished" podID="af895778-508d-466d-b7af-8747c06211b9" containerID="848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507" exitCode=0 Jul 29 19:29:31.626544 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.626506 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" Jul 29 19:29:31.626731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.626539 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" event={"ID":"af895778-508d-466d-b7af-8747c06211b9","Type":"ContainerDied","Data":"848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507"} Jul 29 19:29:31.626731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.626591 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp" event={"ID":"af895778-508d-466d-b7af-8747c06211b9","Type":"ContainerDied","Data":"d5946134de13c7dd479d931152025b900924c30ae7eaa132d22c7e39ca04d21e"} Jul 29 19:29:31.626731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.626612 2574 scope.go:117] "RemoveContainer" containerID="848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507" Jul 29 19:29:31.639407 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.639386 2574 scope.go:117] "RemoveContainer" containerID="47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a" Jul 29 19:29:31.647419 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.647402 2574 scope.go:117] "RemoveContainer" containerID="848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507" Jul 29 19:29:31.647685 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:29:31.647661 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507\": container with ID starting with 848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507 not found: ID does not exist" containerID="848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507" Jul 29 19:29:31.647736 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.647693 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507"} err="failed to get container status \"848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507\": rpc error: code = NotFound desc = could not find container \"848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507\": container with ID starting with 848cf95a8b35043038287d32c1f6ac57a4b3503296cb749b493115b11c18e507 not found: ID does not exist" Jul 29 19:29:31.647736 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.647707 2574 scope.go:117] "RemoveContainer" containerID="47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a" Jul 29 19:29:31.647982 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:29:31.647963 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a\": container with ID starting with 47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a not found: ID does not exist" containerID="47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a" Jul 29 19:29:31.648040 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.647988 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a"} err="failed to get container status \"47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a\": rpc error: code = NotFound desc = could not find container \"47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a\": container with ID starting with 47f075ccf3d0933be2e66d077a36223aa0094539bad53d0e4d4e406842ba3a3a not found: ID does not exist" Jul 29 19:29:31.654002 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.653981 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp"] Jul 29 19:29:31.658759 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:31.658737 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-d04442-openshift-default-dc4b65bd8-r5rxp"] Jul 29 19:29:32.768035 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:32.767998 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="af895778-508d-466d-b7af-8747c06211b9" path="/var/lib/kubelet/pods/af895778-508d-466d-b7af-8747c06211b9/volumes" Jul 29 19:29:36.924782 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:36.924743 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn"] Jul 29 19:29:36.925301 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:36.925019 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" podUID="dc4d0ac2-81ad-4d94-9edf-b92638f2444b" containerName="payload-processing" containerID="cri-o://cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f" gracePeriod=30 Jul 29 19:29:37.188612 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.188541 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:29:37.274804 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.274775 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-msd4f\" (UniqueName: \"kubernetes.io/projected/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-kube-api-access-msd4f\") pod \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " Jul 29 19:29:37.274985 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.274892 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-plugins-config-volume\") pod \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\" (UID: \"dc4d0ac2-81ad-4d94-9edf-b92638f2444b\") " Jul 29 19:29:37.275252 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.275230 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "dc4d0ac2-81ad-4d94-9edf-b92638f2444b" (UID: "dc4d0ac2-81ad-4d94-9edf-b92638f2444b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:37.277027 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.277009 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-kube-api-access-msd4f" (OuterVolumeSpecName: "kube-api-access-msd4f") pod "dc4d0ac2-81ad-4d94-9edf-b92638f2444b" (UID: "dc4d0ac2-81ad-4d94-9edf-b92638f2444b"). InnerVolumeSpecName "kube-api-access-msd4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:37.375859 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.375823 2574 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-plugins-config-volume\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:37.375859 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.375853 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msd4f\" (UniqueName: \"kubernetes.io/projected/dc4d0ac2-81ad-4d94-9edf-b92638f2444b-kube-api-access-msd4f\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:37.657028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.656906 2574 generic.go:356] "Generic (PLEG): container finished" podID="dc4d0ac2-81ad-4d94-9edf-b92638f2444b" containerID="cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f" exitCode=0 Jul 29 19:29:37.657028 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.656980 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" Jul 29 19:29:37.657221 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.657026 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" event={"ID":"dc4d0ac2-81ad-4d94-9edf-b92638f2444b","Type":"ContainerDied","Data":"cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f"} Jul 29 19:29:37.657221 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.657075 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn" event={"ID":"dc4d0ac2-81ad-4d94-9edf-b92638f2444b","Type":"ContainerDied","Data":"b465213a798152d2efef03caff27392f0598a3fbf9f0e1460299a4946801d6dd"} Jul 29 19:29:37.657221 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.657096 2574 scope.go:117] "RemoveContainer" containerID="cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f" Jul 29 19:29:37.666444 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.666425 2574 scope.go:117] "RemoveContainer" containerID="cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f" Jul 29 19:29:37.666699 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:29:37.666682 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f\": container with ID starting with cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f not found: ID does not exist" containerID="cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f" Jul 29 19:29:37.666739 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.666712 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f"} err="failed to get container status \"cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f\": rpc error: code = NotFound desc = could not find container \"cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f\": container with ID starting with cc39fbdcf6bed9975b340c8659379137f9840796244b2c91005643210aa5c65f not found: ID does not exist" Jul 29 19:29:37.684935 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.684899 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn"] Jul 29 19:29:37.690589 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:37.690565 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-422d30-7948cb7f6b-r79tn"] Jul 29 19:29:38.767372 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:38.767339 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4d0ac2-81ad-4d94-9edf-b92638f2444b" path="/var/lib/kubelet/pods/dc4d0ac2-81ad-4d94-9edf-b92638f2444b/volumes" Jul 29 19:29:43.792738 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:43.792653 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg"] Jul 29 19:29:43.793210 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:43.792950 2574 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" containerID="cri-o://ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad" gracePeriod=30 Jul 29 19:29:45.299098 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:45.299062 2574 patch_prober.go:28] interesting pod/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 29 19:29:45.299483 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:45.299136 2574 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 29 19:29:49.061218 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.061151 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:29:49.194693 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194662 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-data\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.194875 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194702 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-credential-socket\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.194875 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194734 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67cc302-b603-4694-aba7-544a3c40232c-istiod-ca-cert\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.194875 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194769 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-istio-token\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.194875 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194799 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-socket\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.194875 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194820 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4kcnz\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-kube-api-access-4kcnz\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.194875 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194845 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67cc302-b603-4694-aba7-544a3c40232c-istio-podinfo\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.195236 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194949 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-envoy\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.195236 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194990 2574 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-certs\") pod \"d67cc302-b603-4694-aba7-544a3c40232c\" (UID: \"d67cc302-b603-4694-aba7-544a3c40232c\") " Jul 29 19:29:49.195236 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.194997 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:49.195236 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.195175 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:49.195439 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.195415 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:49.195495 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.195440 2574 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-credential-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.195495 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.195459 2574 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-socket\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.195593 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.195566 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:49.195847 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.195814 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d67cc302-b603-4694-aba7-544a3c40232c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 29 19:29:49.197392 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.197362 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-kube-api-access-4kcnz" (OuterVolumeSpecName: "kube-api-access-4kcnz") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "kube-api-access-4kcnz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:49.197493 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.197381 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 29 19:29:49.197534 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.197515 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 29 19:29:49.197589 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.197567 2574 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d67cc302-b603-4694-aba7-544a3c40232c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d67cc302-b603-4694-aba7-544a3c40232c" (UID: "d67cc302-b603-4694-aba7-544a3c40232c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 29 19:29:49.296339 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296306 2574 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-data\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.296339 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296332 2574 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d67cc302-b603-4694-aba7-544a3c40232c-istiod-ca-cert\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.296339 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296343 2574 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-istio-token\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.296560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296352 2574 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kcnz\" (UniqueName: \"kubernetes.io/projected/d67cc302-b603-4694-aba7-544a3c40232c-kube-api-access-4kcnz\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.296560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296361 2574 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d67cc302-b603-4694-aba7-544a3c40232c-istio-podinfo\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.296560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296369 2574 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-istio-envoy\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.296560 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.296377 2574 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d67cc302-b603-4694-aba7-544a3c40232c-workload-certs\") on node \"ip-10-0-137-210.ec2.internal\" DevicePath \"\"" Jul 29 19:29:49.718688 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.718651 2574 generic.go:356] "Generic (PLEG): container finished" podID="d67cc302-b603-4694-aba7-544a3c40232c" containerID="ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad" exitCode=0 Jul 29 19:29:49.718688 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.718692 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" event={"ID":"d67cc302-b603-4694-aba7-544a3c40232c","Type":"ContainerDied","Data":"ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad"} Jul 29 19:29:49.718965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.718726 2574 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" Jul 29 19:29:49.718965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.718763 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg" event={"ID":"d67cc302-b603-4694-aba7-544a3c40232c","Type":"ContainerDied","Data":"6f881caaafef7889321ac1fee1062184732d3ec60427506e2822ef84bc9ef328"} Jul 29 19:29:49.718965 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.718786 2574 scope.go:117] "RemoveContainer" containerID="ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad" Jul 29 19:29:49.728155 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.728136 2574 scope.go:117] "RemoveContainer" containerID="c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c" Jul 29 19:29:49.736323 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.736306 2574 scope.go:117] "RemoveContainer" containerID="ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad" Jul 29 19:29:49.736577 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:29:49.736556 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad\": container with ID starting with ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad not found: ID does not exist" containerID="ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad" Jul 29 19:29:49.736662 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.736589 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad"} err="failed to get container status \"ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad\": rpc error: code = NotFound desc = could not find container \"ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad\": container with ID starting with ad83598b576ff89a99d84d726751544815466a1b7c2043d9135ee2e8006b76ad not found: ID does not exist" Jul 29 19:29:49.736662 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.736614 2574 scope.go:117] "RemoveContainer" containerID="c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c" Jul 29 19:29:49.736851 ip-10-0-137-210 kubenswrapper[2574]: E0729 19:29:49.736834 2574 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c\": container with ID starting with c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c not found: ID does not exist" containerID="c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c" Jul 29 19:29:49.736890 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.736860 2574 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c"} err="failed to get container status \"c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c\": rpc error: code = NotFound desc = could not find container \"c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c\": container with ID starting with c641d595cd119e347d64695a2022d4402f34f1cd9fe2777d55cf6daae6cbe56c not found: ID does not exist" Jul 29 19:29:49.741230 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.741209 2574 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg"] Jul 29 19:29:49.745769 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:49.745748 2574 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-422d30-openshift-default-7598f688cb-cpxdg"] Jul 29 19:29:50.392247 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:50.392207 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d7f7957b6-2px5x_81ad53d3-9475-468a-b93d-934074d78159/manager/0.log" Jul 29 19:29:50.507003 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:50.506970 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sk5nz_8228c2e6-58e9-40a1-8f97-4111fd04b661/server/0.log" Jul 29 19:29:50.767755 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:50.767723 2574 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d67cc302-b603-4694-aba7-544a3c40232c" path="/var/lib/kubelet/pods/d67cc302-b603-4694-aba7-544a3c40232c/volumes" Jul 29 19:29:50.769549 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:50.769529 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-296sp_c4496f32-73ba-4e03-ba5f-92c628428d13/copy-manifests/0.log" Jul 29 19:29:50.776632 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:50.776602 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-296sp_c4496f32-73ba-4e03-ba5f-92c628428d13/manager/0.log" Jul 29 19:29:51.440588 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:51.440552 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6985f9bc9c-hgdqb_41337600-2a96-48de-ae81-bce4a122a3ff/maas-api/0.log" Jul 29 19:29:53.377529 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:53.377491 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-q6spr_bf35b243-3d17-4d07-9f3f-3165aa313ae1/manager/0.log" Jul 29 19:29:53.729847 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:53.729797 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-trfbj_a69c9973-8278-46f0-ab9d-d9b3b97cd4a3/registry-server/0.log" Jul 29 19:29:53.862545 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:53.862512 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-w8cpm_dceb4e5e-70d4-4aa1-93e1-426305c5350b/manager/0.log" Jul 29 19:29:54.467907 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:54.467870 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk_4e5b83d0-4223-4bf4-85fd-53884cb90808/istio-proxy/0.log" Jul 29 19:29:54.590455 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:54.590422 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g4g8l_93ee3ec0-391d-4b11-ac70-c5a393d66810/discovery/0.log" Jul 29 19:29:54.823116 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:54.823027 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-tpd98_f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6/kube-auth-proxy/0.log" Jul 29 19:29:54.950809 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:54.950774 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/1.log" Jul 29 19:29:55.077991 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:55.077887 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jqmvd_f96d6bae-87b7-4c81-9ca1-fe2cdd75db18/payload-pre-processing/0.log" Jul 29 19:29:55.203970 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:55.203940 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-bps2k_8b2eeae5-b163-4f47-9581-096c74aaafca/payload-processing/0.log" Jul 29 19:29:55.825781 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:55.825749 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-mt7df_57fe6ad0-6337-4ad8-a6ad-056afc8268bd/main/0.log" Jul 29 19:29:56.068611 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:29:56.068575 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f4b6c9_2b2f994a-3752-4192-89a3-347d5154cd20/main/0.log" Jul 29 19:30:05.744044 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:05.743986 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lzz4p_aee5f3b0-2217-4539-b351-afec36c4fb22/global-pull-secret-syncer/0.log" Jul 29 19:30:06.018197 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:06.018099 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-qhf4c_2ad76cc7-9aa2-4664-bcb1-c00ffbbc3af0/konnectivity-agent/0.log" Jul 29 19:30:06.048340 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:06.048304 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-210.ec2.internal_37538435c645a6e5590e800e1cbd349a/haproxy/0.log" Jul 29 19:30:11.205673 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:11.205638 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-q6spr_bf35b243-3d17-4d07-9f3f-3165aa313ae1/manager/0.log" Jul 29 19:30:11.336347 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:11.336313 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-trfbj_a69c9973-8278-46f0-ab9d-d9b3b97cd4a3/registry-server/0.log" Jul 29 19:30:11.602163 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:11.602071 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-w8cpm_dceb4e5e-70d4-4aa1-93e1-426305c5350b/manager/0.log" Jul 29 19:30:13.630878 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:13.630842 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-nwttw_8722654c-a11e-4df1-87f2-fa6137ad83e4/kube-state-metrics/0.log" Jul 29 19:30:13.649317 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:13.649287 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-nwttw_8722654c-a11e-4df1-87f2-fa6137ad83e4/kube-rbac-proxy-main/0.log" Jul 29 19:30:13.671815 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:13.671735 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-nwttw_8722654c-a11e-4df1-87f2-fa6137ad83e4/kube-rbac-proxy-self/0.log" Jul 29 19:30:13.772094 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:13.772065 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9qj4p_1b347548-f005-4036-bed6-d91a0815d58d/node-exporter/0.log" Jul 29 19:30:13.797781 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:13.797756 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9qj4p_1b347548-f005-4036-bed6-d91a0815d58d/kube-rbac-proxy/0.log" Jul 29 19:30:13.828445 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:13.828409 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9qj4p_1b347548-f005-4036-bed6-d91a0815d58d/init-textfile/0.log" Jul 29 19:30:14.124988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.124898 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/prometheus/0.log" Jul 29 19:30:14.156820 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.156783 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/config-reloader/0.log" Jul 29 19:30:14.185476 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.185448 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/thanos-sidecar/0.log" Jul 29 19:30:14.216666 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.216628 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/kube-rbac-proxy-web/0.log" Jul 29 19:30:14.242219 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.242180 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/kube-rbac-proxy/0.log" Jul 29 19:30:14.270770 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.270739 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/kube-rbac-proxy-thanos/0.log" Jul 29 19:30:14.299062 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:14.299037 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_b233f2b3-97d3-43b4-b28c-73bfa117757a/init-config-reloader/0.log" Jul 29 19:30:15.587585 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587547 2574 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52"] Jul 29 19:30:15.587988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587946 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" Jul 29 19:30:15.587988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587960 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" Jul 29 19:30:15.587988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587972 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42357943-4a3a-4476-8c16-a0d9e28c6a47" containerName="payload-processing" Jul 29 19:30:15.587988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587978 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="42357943-4a3a-4476-8c16-a0d9e28c6a47" containerName="payload-processing" Jul 29 19:30:15.587988 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587990 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.587996 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588002 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588007 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588013 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f733a38f-98f0-4f57-b01f-816a953fa485" containerName="payload-pre-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588018 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="f733a38f-98f0-4f57-b01f-816a953fa485" containerName="payload-pre-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588028 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4d0ac2-81ad-4d94-9edf-b92638f2444b" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588033 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4d0ac2-81ad-4d94-9edf-b92638f2444b" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588041 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588046 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588055 2574 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272b63a2-c286-45a8-885e-076c852f9398" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588060 2574 state_mem.go:107] "Deleted CPUSet assignment" podUID="272b63a2-c286-45a8-885e-076c852f9398" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588115 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4d0ac2-81ad-4d94-9edf-b92638f2444b" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588122 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="42357943-4a3a-4476-8c16-a0d9e28c6a47" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588129 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="af895778-508d-466d-b7af-8747c06211b9" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588135 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588141 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588146 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5df3d34-8c16-4655-a8f5-1e3ffc25c578" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588151 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="d67cc302-b603-4694-aba7-544a3c40232c" containerName="istio-proxy" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588157 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="272b63a2-c286-45a8-885e-076c852f9398" containerName="payload-processing" Jul 29 19:30:15.588162 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.588163 2574 memory_manager.go:368] "RemoveStaleState removing state" podUID="f733a38f-98f0-4f57-b01f-816a953fa485" containerName="payload-pre-processing" Jul 29 19:30:15.591310 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.591289 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.594431 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.594412 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmkl2\"/\"kube-root-ca.crt\"" Jul 29 19:30:15.594599 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.594584 2574 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-kmkl2\"/\"default-dockercfg-qtdw2\"" Jul 29 19:30:15.594930 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.594899 2574 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-kmkl2\"/\"openshift-service-ca.crt\"" Jul 29 19:30:15.607447 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.607426 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52"] Jul 29 19:30:15.745274 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.745242 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-proc\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.745457 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.745288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbhwz\" (UniqueName: \"kubernetes.io/projected/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-kube-api-access-qbhwz\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.745457 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.745323 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-lib-modules\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.745457 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.745379 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-sys\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.745457 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.745435 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-podres\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846079 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.845985 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-proc\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846079 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846037 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbhwz\" (UniqueName: \"kubernetes.io/projected/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-kube-api-access-qbhwz\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846079 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846067 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-lib-modules\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846095 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-sys\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846107 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-proc\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846166 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-podres\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846186 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-sys\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846269 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-podres\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.846375 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.846270 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-lib-modules\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.855281 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.855259 2574 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbhwz\" (UniqueName: \"kubernetes.io/projected/7e9558d6-2511-40ff-b8a4-b7bfdb37c809-kube-api-access-qbhwz\") pod \"perf-node-gather-daemonset-hgc52\" (UID: \"7e9558d6-2511-40ff-b8a4-b7bfdb37c809\") " pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:15.902166 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:15.902114 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:16.040438 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:16.040414 2574 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52"] Jul 29 19:30:16.042464 ip-10-0-137-210 kubenswrapper[2574]: W0729 19:30:16.042439 2574 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod7e9558d6_2511_40ff_b8a4_b7bfdb37c809.slice/crio-857f3c327aefdc730d1cb98dc81f1ac93e12a35c6b5bce931754fbd6146a1cc8 WatchSource:0}: Error finding container 857f3c327aefdc730d1cb98dc81f1ac93e12a35c6b5bce931754fbd6146a1cc8: Status 404 returned error can't find the container with id 857f3c327aefdc730d1cb98dc81f1ac93e12a35c6b5bce931754fbd6146a1cc8 Jul 29 19:30:16.855887 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:16.855845 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" event={"ID":"7e9558d6-2511-40ff-b8a4-b7bfdb37c809","Type":"ContainerStarted","Data":"5396f31157ea50415aaeabeafef54c644052f778fe5933d8ec62c4038b4e33d4"} Jul 29 19:30:16.856348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:16.855910 2574 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" event={"ID":"7e9558d6-2511-40ff-b8a4-b7bfdb37c809","Type":"ContainerStarted","Data":"857f3c327aefdc730d1cb98dc81f1ac93e12a35c6b5bce931754fbd6146a1cc8"} Jul 29 19:30:16.856348 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:16.856084 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:16.873882 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:16.873834 2574 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" podStartSLOduration=1.8738187819999998 podStartE2EDuration="1.873818782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-29 19:30:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-29 19:30:16.873678446 +0000 UTC m=+3356.586589936" watchObservedRunningTime="2026-07-29 19:30:16.873818782 +0000 UTC m=+3356.586730271" Jul 29 19:30:16.959958 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:16.959905 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-689bcc974d-sc47h_c26d0e1e-95a0-484c-9914-f11c3143f61a/console/0.log" Jul 29 19:30:18.681956 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:18.681906 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fq5r_2cf63bdd-f881-4c6e-9a12-65930772fa1d/dns/0.log" Jul 29 19:30:18.706899 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:18.706867 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-8fq5r_2cf63bdd-f881-4c6e-9a12-65930772fa1d/kube-rbac-proxy/0.log" Jul 29 19:30:18.900310 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:18.900282 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-zp82l_fb2c2975-b7bc-489d-85cd-831a7d745acc/dns-node-resolver/0.log" Jul 29 19:30:19.456634 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:19.456595 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5f6d6d5c85-6hnmr_0c5cbc72-89f0-47ca-aa3f-bb66c74d47db/registry/0.log" Jul 29 19:30:19.481821 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:19.481781 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lb7g7_d8b39bd2-2b35-4acd-bb0d-8f70ef7af169/node-ca/0.log" Jul 29 19:30:20.501152 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:20.501094 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk_4e5b83d0-4223-4bf4-85fd-53884cb90808/istio-proxy/0.log" Jul 29 19:30:20.796693 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:20.796599 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g4g8l_93ee3ec0-391d-4b11-ac70-c5a393d66810/discovery/0.log" Jul 29 19:30:20.846257 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:20.846227 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-tpd98_f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6/kube-auth-proxy/0.log" Jul 29 19:30:20.878311 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:20.878282 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/0.log" Jul 29 19:30:20.936113 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:20.936076 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/1.log" Jul 29 19:30:20.977511 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:20.977480 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jqmvd_f96d6bae-87b7-4c81-9ca1-fe2cdd75db18/payload-pre-processing/0.log" Jul 29 19:30:21.020297 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:21.020136 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-bps2k_8b2eeae5-b163-4f47-9581-096c74aaafca/payload-processing/0.log" Jul 29 19:30:21.897401 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:21.897368 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gn8s2_93db8590-3846-473d-a8e1-d68e33613e1a/serve-healthcheck-canary/0.log" Jul 29 19:30:22.450334 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:22.450303 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n967v_eaf8ea8d-12fc-4eba-bf4a-de072713200a/kube-rbac-proxy/0.log" Jul 29 19:30:22.478637 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:22.478612 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n967v_eaf8ea8d-12fc-4eba-bf4a-de072713200a/exporter/0.log" Jul 29 19:30:22.505839 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:22.505810 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-n967v_eaf8ea8d-12fc-4eba-bf4a-de072713200a/extractor/0.log" Jul 29 19:30:22.871473 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:22.871397 2574 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-kmkl2/perf-node-gather-daemonset-hgc52" Jul 29 19:30:25.969134 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:25.969100 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-d7f7957b6-2px5x_81ad53d3-9475-468a-b93d-934074d78159/manager/0.log" Jul 29 19:30:25.993473 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:25.993443 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-sk5nz_8228c2e6-58e9-40a1-8f97-4111fd04b661/server/0.log" Jul 29 19:30:26.055743 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.055707 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk_4e5b83d0-4223-4bf4-85fd-53884cb90808/istio-proxy/0.log" Jul 29 19:30:26.136938 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.136875 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-296sp_c4496f32-73ba-4e03-ba5f-92c628428d13/manager/0.log" Jul 29 19:30:26.165796 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.165766 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-76c6f8f9b-296sp_c4496f32-73ba-4e03-ba5f-92c628428d13/copy-manifests/0.log" Jul 29 19:30:26.331887 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.331801 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g4g8l_93ee3ec0-391d-4b11-ac70-c5a393d66810/discovery/0.log" Jul 29 19:30:26.406240 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.406207 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-tpd98_f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6/kube-auth-proxy/0.log" Jul 29 19:30:26.435665 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.435633 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/0.log" Jul 29 19:30:26.489969 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.489935 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/1.log" Jul 29 19:30:26.527821 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.527789 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jqmvd_f96d6bae-87b7-4c81-9ca1-fe2cdd75db18/payload-pre-processing/0.log" Jul 29 19:30:26.559568 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:26.559537 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-bps2k_8b2eeae5-b163-4f47-9581-096c74aaafca/payload-processing/0.log" Jul 29 19:30:27.714381 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:27.714330 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-fk8ld_97a86d73-1fb1-42f4-88c7-055fb582c419/openshift-lws-operator/0.log" Jul 29 19:30:27.912414 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:27.912381 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-29vwk_4e5b83d0-4223-4bf4-85fd-53884cb90808/istio-proxy/0.log" Jul 29 19:30:28.201731 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:28.201701 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-g4g8l_93ee3ec0-391d-4b11-ac70-c5a393d66810/discovery/0.log" Jul 29 19:30:28.231533 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:28.231488 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-54cd75bc79-tpd98_f5bf760c-11ea-46c1-8ad1-43bbe8bc83e6/kube-auth-proxy/0.log" Jul 29 19:30:28.255459 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:28.255422 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/0.log" Jul 29 19:30:28.313046 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:28.313009 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-wlxnw_6dcf6e36-4709-4f76-8190-0ed9dfdd4db8/istio-proxy/1.log" Jul 29 19:30:28.354550 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:28.354516 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-jqmvd_f96d6bae-87b7-4c81-9ca1-fe2cdd75db18/payload-pre-processing/0.log" Jul 29 19:30:28.390882 ip-10-0-137-210 kubenswrapper[2574]: I0729 19:30:28.390852 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5cc6cd444b-bps2k_8b2eeae5-b163-4f47-9581-096c74aaafca/payload-processing/0.log"