Jul 23 11:35:50.672135 ip-10-0-141-203 systemd[1]: Starting Kubernetes Kubelet... Jul 23 11:35:51.088910 ip-10-0-141-203 kubenswrapper[2554]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:35:51.088910 ip-10-0-141-203 kubenswrapper[2554]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 23 11:35:51.088910 ip-10-0-141-203 kubenswrapper[2554]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:35:51.088910 ip-10-0-141-203 kubenswrapper[2554]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 23 11:35:51.088910 ip-10-0-141-203 kubenswrapper[2554]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 23 11:35:51.090278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.090129 2554 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 23 11:35:51.095431 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095398 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:51.095431 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095425 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:51.095431 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095430 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:51.095431 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095434 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:51.095431 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095438 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:51.095431 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095442 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095447 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095451 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095456 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095462 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095466 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095470 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095474 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095477 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095481 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095484 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095488 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095491 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095495 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095499 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095505 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095510 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095515 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095519 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:51.095819 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095523 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095527 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095531 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095535 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095539 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095547 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095552 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095556 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095560 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095564 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095568 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095572 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095576 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095580 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095584 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095589 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095593 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095598 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095602 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:51.096583 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095606 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095611 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095616 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095620 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095624 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095628 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095632 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095639 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095644 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095648 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095652 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095656 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095661 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095665 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095669 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095673 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095678 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095682 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095687 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095691 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:51.097360 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095695 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095699 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095703 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095707 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095711 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095715 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095719 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095726 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095732 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095737 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095741 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095766 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095771 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095775 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095779 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095783 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095787 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095793 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095797 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:51.097893 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095802 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095806 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095810 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095814 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095817 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095821 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095825 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095829 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095833 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095837 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095841 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095846 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095850 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095855 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095859 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095863 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.095867 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097094 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097105 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097110 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:51.098641 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097116 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097120 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097124 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097128 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097134 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097138 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097142 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097147 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097151 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097155 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097160 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097165 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097169 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097174 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097178 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097182 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097186 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097190 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097194 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097199 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:51.099494 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097203 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097207 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097211 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097216 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097220 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097224 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097229 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097233 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097238 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097242 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097246 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097250 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097255 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097259 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097262 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097267 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097271 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.097276 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098214 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098229 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:51.100383 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098235 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098239 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098244 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098249 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098264 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098269 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098274 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098278 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098283 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098287 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098291 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098295 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098304 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098308 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098313 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098319 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098323 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098328 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098335 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:51.101279 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098341 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098346 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098350 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098355 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098360 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098365 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098370 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098375 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098381 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098386 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098390 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098394 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098399 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098403 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098410 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098415 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098420 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098425 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098434 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098439 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:51.101903 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098443 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098447 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098452 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098456 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098460 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098467 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098472 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098477 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098481 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098486 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098490 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098495 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098498 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098503 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098507 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098511 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.098515 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099199 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099208 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:51.102607 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099213 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099220 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099227 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099231 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099236 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099240 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099244 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099248 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099252 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099257 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099261 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099266 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099271 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099275 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099279 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099283 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099287 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099291 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099295 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:51.103102 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099302 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099307 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099313 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099318 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099322 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099327 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099331 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099336 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099340 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099345 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099349 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099355 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099359 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099363 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099368 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099372 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099377 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099381 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099386 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099390 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:51.103609 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099395 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099399 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099405 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099410 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099415 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099421 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099426 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099431 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099437 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099441 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099445 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099449 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099453 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099458 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099462 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099467 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099471 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099475 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099479 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:51.104214 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099483 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099487 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099491 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099495 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099501 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099505 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099509 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099514 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099518 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099522 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099525 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099529 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099533 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099537 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099542 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099546 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099550 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099554 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099558 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:51.104681 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099563 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099567 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099571 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099575 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099579 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099584 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099588 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099592 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099598 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099603 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099607 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099611 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099615 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099620 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099624 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099628 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099632 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099637 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099641 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.099645 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:51.105173 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099783 2554 flags.go:64] FLAG: --address="0.0.0.0" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099796 2554 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099807 2554 flags.go:64] FLAG: --anonymous-auth="true" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099814 2554 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099820 2554 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099826 2554 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099833 2554 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099840 2554 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099845 2554 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099850 2554 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099855 2554 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099861 2554 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099866 2554 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099872 2554 flags.go:64] FLAG: --cgroup-root="" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099876 2554 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099881 2554 flags.go:64] FLAG: --client-ca-file="" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099892 2554 flags.go:64] FLAG: --cloud-provider="external" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099897 2554 flags.go:64] FLAG: --cluster-dns="[]" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099904 2554 flags.go:64] FLAG: --cluster-domain="" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099908 2554 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099913 2554 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099918 2554 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099923 2554 flags.go:64] FLAG: --container-log-max-files="5" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099929 2554 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 23 11:35:51.105686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099934 2554 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099939 2554 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099944 2554 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099949 2554 flags.go:64] FLAG: --contention-profiling="false" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099954 2554 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099959 2554 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099964 2554 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099969 2554 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099977 2554 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099982 2554 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099987 2554 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099992 2554 flags.go:64] FLAG: --enable-load-reader="false" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.099996 2554 flags.go:64] FLAG: --enable-server="true" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100001 2554 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100009 2554 flags.go:64] FLAG: --event-burst="100" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100013 2554 flags.go:64] FLAG: --event-qps="50" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100018 2554 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100023 2554 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100028 2554 flags.go:64] FLAG: --eviction-hard="" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100034 2554 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100039 2554 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100044 2554 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100049 2554 flags.go:64] FLAG: --eviction-soft="" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100054 2554 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100059 2554 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 23 11:35:51.106278 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100064 2554 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100072 2554 flags.go:64] FLAG: --experimental-mounter-path="" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100076 2554 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100081 2554 flags.go:64] FLAG: --fail-swap-on="true" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100085 2554 flags.go:64] FLAG: --feature-gates="" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100091 2554 flags.go:64] FLAG: --file-check-frequency="20s" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100096 2554 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100101 2554 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100106 2554 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100112 2554 flags.go:64] FLAG: --healthz-port="10248" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100117 2554 flags.go:64] FLAG: --help="false" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100122 2554 flags.go:64] FLAG: --hostname-override="ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100127 2554 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100131 2554 flags.go:64] FLAG: --http-check-frequency="20s" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100136 2554 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100142 2554 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100148 2554 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100152 2554 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100157 2554 flags.go:64] FLAG: --image-service-endpoint="" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100161 2554 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100166 2554 flags.go:64] FLAG: --kube-api-burst="100" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100170 2554 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100175 2554 flags.go:64] FLAG: --kube-api-qps="50" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100180 2554 flags.go:64] FLAG: --kube-reserved="" Jul 23 11:35:51.106930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100185 2554 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100189 2554 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100194 2554 flags.go:64] FLAG: --kubelet-cgroups="" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100199 2554 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100203 2554 flags.go:64] FLAG: --lock-file="" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100208 2554 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100215 2554 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100221 2554 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100230 2554 flags.go:64] FLAG: --log-json-split-stream="false" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100236 2554 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100241 2554 flags.go:64] FLAG: --log-text-split-stream="false" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100245 2554 flags.go:64] FLAG: --logging-format="text" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100250 2554 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100255 2554 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100259 2554 flags.go:64] FLAG: --manifest-url="" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100264 2554 flags.go:64] FLAG: --manifest-url-header="" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100271 2554 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100279 2554 flags.go:64] FLAG: --max-open-files="1000000" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100285 2554 flags.go:64] FLAG: --max-pods="110" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100290 2554 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100294 2554 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100298 2554 flags.go:64] FLAG: --memory-manager-policy="None" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100304 2554 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100309 2554 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100313 2554 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 23 11:35:51.107603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100318 2554 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100330 2554 flags.go:64] FLAG: --node-status-max-images="50" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100334 2554 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100339 2554 flags.go:64] FLAG: --oom-score-adj="-999" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100344 2554 flags.go:64] FLAG: --pod-cidr="" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100349 2554 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100358 2554 flags.go:64] FLAG: --pod-manifest-path="" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100362 2554 flags.go:64] FLAG: --pod-max-pids="-1" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100367 2554 flags.go:64] FLAG: --pods-per-core="0" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100372 2554 flags.go:64] FLAG: --port="10250" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100377 2554 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100381 2554 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03968e3db5f139824" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100386 2554 flags.go:64] FLAG: --qos-reserved="" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100391 2554 flags.go:64] FLAG: --read-only-port="10255" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100396 2554 flags.go:64] FLAG: --register-node="true" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100401 2554 flags.go:64] FLAG: --register-with-taints="" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100408 2554 flags.go:64] FLAG: --registry-burst="10" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100413 2554 flags.go:64] FLAG: --registry-qps="5" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100418 2554 flags.go:64] FLAG: --reserved-cpus="" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100422 2554 flags.go:64] FLAG: --reserved-memory="" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100429 2554 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100434 2554 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100439 2554 flags.go:64] FLAG: --rotate-certificates="false" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100444 2554 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100449 2554 flags.go:64] FLAG: --runonce="false" Jul 23 11:35:51.108266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100455 2554 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100461 2554 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100466 2554 flags.go:64] FLAG: --seccomp-default="false" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100471 2554 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100476 2554 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100481 2554 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100486 2554 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100490 2554 flags.go:64] FLAG: --storage-driver-password="root" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100495 2554 flags.go:64] FLAG: --storage-driver-secure="false" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100499 2554 flags.go:64] FLAG: --storage-driver-table="stats" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100504 2554 flags.go:64] FLAG: --storage-driver-user="root" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100509 2554 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100514 2554 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100519 2554 flags.go:64] FLAG: --system-cgroups="" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100523 2554 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100531 2554 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100536 2554 flags.go:64] FLAG: --tls-cert-file="" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100540 2554 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100546 2554 flags.go:64] FLAG: --tls-min-version="" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100550 2554 flags.go:64] FLAG: --tls-private-key-file="" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100555 2554 flags.go:64] FLAG: --topology-manager-policy="none" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100559 2554 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100565 2554 flags.go:64] FLAG: --topology-manager-scope="container" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100572 2554 flags.go:64] FLAG: --v="2" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100578 2554 flags.go:64] FLAG: --version="false" Jul 23 11:35:51.108901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100585 2554 flags.go:64] FLAG: --vmodule="" Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100590 2554 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.100595 2554 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100802 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100809 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100814 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100818 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100824 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100831 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100836 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100841 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100846 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100851 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100855 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100860 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100864 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100869 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100873 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100877 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:51.109553 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100881 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100886 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100890 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100893 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100897 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100902 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100906 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100910 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100914 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100919 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100923 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100930 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100934 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100938 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100942 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100946 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100950 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100954 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100959 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100963 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:51.110053 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100969 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100973 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100977 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100981 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100987 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100992 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.100997 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101001 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101005 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101009 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101013 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101017 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101022 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101026 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101030 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101034 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101038 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101045 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101050 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:51.110555 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101055 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101060 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101064 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101068 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101074 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101081 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101085 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101090 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101094 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101099 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101103 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101109 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101113 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101119 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101123 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101127 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101131 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101136 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101140 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:51.111056 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101144 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101148 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101152 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101156 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101161 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101165 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101169 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101173 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101177 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101181 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101185 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101189 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101193 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101197 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101201 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101206 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101210 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101215 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101219 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101225 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:51.111527 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101228 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101233 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101236 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101240 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.101244 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.101258 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.108876 2554 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.108895 2554 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.108922 2554 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.108933 2554 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109003 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109009 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109013 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109016 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:51.112047 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109019 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109023 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109026 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109029 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109031 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109034 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109037 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109039 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109042 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109045 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109048 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109050 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109053 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109056 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109058 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109061 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109064 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109070 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109075 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109079 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:51.112405 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109082 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109085 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109087 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109090 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109094 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109097 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109099 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109102 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109106 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109110 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109113 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109116 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109119 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109123 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109126 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109129 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109132 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109134 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109137 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:51.112909 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109139 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109142 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109147 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109151 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109156 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109159 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109162 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109165 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109167 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109170 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109172 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109176 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109178 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109181 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109183 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109185 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109188 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109191 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109194 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109196 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:51.113387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109199 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109201 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109204 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109206 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109209 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109211 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109214 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109216 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109220 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109224 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109229 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109233 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109237 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109239 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109242 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109246 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109249 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109252 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109254 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:51.113886 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109257 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109259 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109264 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109267 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109269 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109272 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109275 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109277 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109279 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109282 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109285 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109288 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109290 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109293 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109296 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109299 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:51.114343 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109303 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.109312 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109435 2554 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109441 2554 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109444 2554 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109447 2554 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109450 2554 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109452 2554 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109456 2554 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109461 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109466 2554 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109470 2554 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109473 2554 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109477 2554 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109481 2554 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 23 11:35:51.114724 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109484 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109486 2554 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109489 2554 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109492 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109495 2554 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109497 2554 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109500 2554 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109503 2554 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109505 2554 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109508 2554 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109511 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109513 2554 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109516 2554 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109519 2554 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109521 2554 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109524 2554 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109526 2554 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109529 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109532 2554 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109537 2554 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 23 11:35:51.115194 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109542 2554 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109546 2554 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109548 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109551 2554 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109553 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109556 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109559 2554 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109561 2554 feature_gate.go:328] unrecognized feature gate: Example2 Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109565 2554 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109568 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109571 2554 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109574 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109576 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109580 2554 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109582 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109584 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109587 2554 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109590 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109592 2554 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 23 11:35:51.115684 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109594 2554 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109597 2554 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109600 2554 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109602 2554 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109605 2554 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109608 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109613 2554 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109618 2554 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109622 2554 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109626 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109628 2554 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109631 2554 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109633 2554 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109636 2554 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109638 2554 feature_gate.go:328] unrecognized feature gate: Example Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109641 2554 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109643 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109646 2554 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109648 2554 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 23 11:35:51.116175 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109651 2554 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109653 2554 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109655 2554 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109658 2554 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109660 2554 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109662 2554 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109665 2554 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109667 2554 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109670 2554 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109672 2554 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109674 2554 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109677 2554 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109680 2554 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109682 2554 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109685 2554 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109689 2554 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109694 2554 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109698 2554 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109702 2554 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109705 2554 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 23 11:35:51.116628 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109708 2554 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109710 2554 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109712 2554 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109715 2554 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109717 2554 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109721 2554 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109724 2554 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:51.109727 2554 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.109732 2554 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.109893 2554 server.go:962] "Client rotation is on, will bootstrap in background" Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.112147 2554 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.113474 2554 server.go:1019] "Starting client certificate rotation" Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.113590 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:35:51.117154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.113633 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 23 11:35:51.117480 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.116713 2554 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:35:51.119985 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.119952 2554 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 23 11:35:51.133215 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.133186 2554 log.go:25] "Validated CRI v1 runtime API" Jul 23 11:35:51.135453 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.135433 2554 log.go:25] "Validated CRI v1 image API" Jul 23 11:35:51.136303 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.136278 2554 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 23 11:35:51.138856 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.138826 2554 fs.go:135] Filesystem UUIDs: map[6b58a38d-7b84-4d30-b1b6-2e1f5afc46b6:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 be5af8b6-3a0b-419a-b0c4-ea754aeb8140:/dev/nvme0n1p3] Jul 23 11:35:51.138922 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.138854 2554 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 23 11:35:51.144344 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.144211 2554 manager.go:216] Machine: {Timestamp:2026-07-23 11:35:51.14300197 +0000 UTC m=+0.347202639 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3095271 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec288aed471f2436b61a818e966da8d8 SystemUUID:ec288aed-471f-2436-b61a-818e966da8d8 BootID:cae0db53-17a8-4c00-9e90-a7e4eec2cc1b Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ab:21:01:47:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ab:21:01:47:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:22:78:f9:58:fa Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 23 11:35:51.144344 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.144332 2554 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 23 11:35:51.144478 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.144431 2554 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 23 11:35:51.144865 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.144846 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:35:51.145859 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.145822 2554 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 23 11:35:51.146031 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.145866 2554 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-203.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 23 11:35:51.146551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.146538 2554 topology_manager.go:138] "Creating topology manager with none policy" Jul 23 11:35:51.146603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.146556 2554 container_manager_linux.go:309] "Creating device plugin manager" Jul 23 11:35:51.146603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.146567 2554 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:35:51.146603 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.146587 2554 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 23 11:35:51.147625 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.147611 2554 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 23 11:35:51.151337 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151317 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:35:51.151486 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151473 2554 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 23 11:35:51.151556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151518 2554 kubelet.go:486] "Attempting to sync node with API server" Jul 23 11:35:51.151556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151533 2554 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 23 11:35:51.151556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151550 2554 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 23 11:35:51.151674 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151566 2554 kubelet.go:398] "Adding apiserver pod source" Jul 23 11:35:51.151674 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.151579 2554 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 23 11:35:51.152027 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.152011 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:35:51.152106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.152035 2554 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 23 11:35:51.154450 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.154434 2554 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 23 11:35:51.156170 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.156153 2554 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 23 11:35:51.156229 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.156186 2554 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 23 11:35:51.157531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157512 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 23 11:35:51.157578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157547 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 23 11:35:51.157578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157557 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 23 11:35:51.157578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157565 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 23 11:35:51.157578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157578 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157585 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157597 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157603 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157610 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157615 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157623 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 23 11:35:51.157680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.157629 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 23 11:35:51.159135 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.159122 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 23 11:35:51.159135 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.159134 2554 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 23 11:35:51.162422 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.162398 2554 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-203.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 23 11:35:51.162830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.162794 2554 server.go:1268] "Started kubelet" Jul 23 11:35:51.163440 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.163400 2554 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 23 11:35:51.163560 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.163423 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-203.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 23 11:35:51.164586 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.163491 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 23 11:35:51.164590 ip-10-0-141-203 systemd[1]: Started Kubernetes Kubelet. Jul 23 11:35:51.165727 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.165566 2554 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 23 11:35:51.165727 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.165695 2554 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 23 11:35:51.166656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.166636 2554 server.go:310] "Adding debug handlers to kubelet server" Jul 23 11:35:51.168344 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.168327 2554 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 23 11:35:51.170028 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.170008 2554 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 23 11:35:51.170266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.170251 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7wtbd" Jul 23 11:35:51.173304 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.172396 2554 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-203.ec2.internal.18c4e81a1ad85e61 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-203.ec2.internal,UID:ip-10-0-141-203.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-203.ec2.internal,},FirstTimestamp:2026-07-23 11:35:51.162740321 +0000 UTC m=+0.366940978,LastTimestamp:2026-07-23 11:35:51.162740321 +0000 UTC m=+0.366940978,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-203.ec2.internal,}" Jul 23 11:35:51.173913 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.173891 2554 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 23 11:35:51.173913 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.173910 2554 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 23 11:35:51.174105 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.174086 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:51.174649 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.174631 2554 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 23 11:35:51.174733 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.174653 2554 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 23 11:35:51.174733 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.174633 2554 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 23 11:35:51.174925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.174784 2554 reconstruct.go:97] "Volume reconstruction finished" Jul 23 11:35:51.174925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.174792 2554 reconciler.go:29] "Reconciler: start to sync state" Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175254 2554 factory.go:153] Registering CRI-O factory Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175272 2554 factory.go:223] Registration of the crio container factory successfully Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175316 2554 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175323 2554 factory.go:55] Registering systemd factory Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175328 2554 factory.go:223] Registration of the systemd container factory successfully Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175353 2554 factory.go:103] Registering Raw factory Jul 23 11:35:51.175437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.175365 2554 manager.go:1196] Started watching for new ooms in manager Jul 23 11:35:51.176074 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.176059 2554 manager.go:349] Starting recovery of all containers Jul 23 11:35:51.177989 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.177963 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7wtbd" Jul 23 11:35:51.186387 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.186182 2554 manager.go:354] Recovery completed Jul 23 11:35:51.187633 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.187601 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:51.187926 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.187899 2554 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 23 11:35:51.191150 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.191132 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.191220 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.191130 2554 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-141-203.ec2.internal\" not found" node="ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.193086 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.193065 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.193167 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.193101 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.193167 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.193115 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.194432 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.194093 2554 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 23 11:35:51.194432 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.194109 2554 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 23 11:35:51.194432 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.194130 2554 state_mem.go:36] "Initialized new in-memory state store" Jul 23 11:35:51.194579 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.194452 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.195417 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.195401 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.195476 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.195429 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.195476 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.195439 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.196266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.196253 2554 policy_none.go:49] "None policy: Start" Jul 23 11:35:51.196310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.196271 2554 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 23 11:35:51.196310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.196282 2554 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 23 11:35:51.198342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.198329 2554 policy_none.go:47] "Start" Jul 23 11:35:51.233689 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.233662 2554 manager.go:337] "Starting Device Plugin manager" Jul 23 11:35:51.233869 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.233706 2554 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 23 11:35:51.233869 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.233717 2554 server.go:87] "Starting device plugin registration server" Jul 23 11:35:51.234060 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.234046 2554 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 23 11:35:51.234108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.234064 2554 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 23 11:35:51.234151 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.234134 2554 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 23 11:35:51.234232 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.234219 2554 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 23 11:35:51.234286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.234233 2554 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 23 11:35:51.234849 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.234824 2554 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 23 11:35:51.234944 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.234877 2554 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:51.335247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.335205 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.337379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.336333 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.337379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.336365 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.337379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.336375 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.337379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.336401 2554 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.346501 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.346425 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 23 11:35:51.347817 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.347800 2554 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 23 11:35:51.347872 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.347843 2554 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 23 11:35:51.347872 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.347866 2554 kubelet.go:2448] "Starting kubelet main sync loop" Jul 23 11:35:51.347962 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.347912 2554 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 23 11:35:51.348190 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.348171 2554 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.348305 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.348196 2554 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-203.ec2.internal\": node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:51.350273 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.350247 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:51.410080 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.410048 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:51.448163 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.448117 2554 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal"] Jul 23 11:35:51.448259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.448220 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.449802 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.449784 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.449876 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.449818 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.449876 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.449828 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.451956 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.451935 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.452069 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.452052 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.452123 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.452101 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.452790 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.452775 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.452883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.452805 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.452883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.452815 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.453657 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.453639 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.453733 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.453669 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.453733 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.453679 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.454937 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.454921 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.455005 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.454950 2554 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 23 11:35:51.455637 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.455615 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientMemory" Jul 23 11:35:51.455637 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.455637 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasNoDiskPressure" Jul 23 11:35:51.455821 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.455651 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeHasSufficientPID" Jul 23 11:35:51.476631 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.476597 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d870a017be816cae8b0075a63393449b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal\" (UID: \"d870a017be816cae8b0075a63393449b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.476631 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.476628 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356e6d0e35ed5731a5b22ac92b6369fe-config\") pod \"kube-apiserver-proxy-ip-10-0-141-203.ec2.internal\" (UID: \"356e6d0e35ed5731a5b22ac92b6369fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.476852 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.476646 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d870a017be816cae8b0075a63393449b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal\" (UID: \"d870a017be816cae8b0075a63393449b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.484832 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.484786 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-203.ec2.internal\" not found" node="ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.489463 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.489441 2554 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-203.ec2.internal\" not found" node="ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.510777 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:51.510734 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:51.577089 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.577059 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356e6d0e35ed5731a5b22ac92b6369fe-config\") pod \"kube-apiserver-proxy-ip-10-0-141-203.ec2.internal\" (UID: \"356e6d0e35ed5731a5b22ac92b6369fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.577089 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.577089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d870a017be816cae8b0075a63393449b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal\" (UID: \"d870a017be816cae8b0075a63393449b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.577225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.577108 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d870a017be816cae8b0075a63393449b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal\" (UID: \"d870a017be816cae8b0075a63393449b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.577225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.577155 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d870a017be816cae8b0075a63393449b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal\" (UID: \"d870a017be816cae8b0075a63393449b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.577225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.577176 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d870a017be816cae8b0075a63393449b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal\" (UID: \"d870a017be816cae8b0075a63393449b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:51.577225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:51.577175 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/356e6d0e35ed5731a5b22ac92b6369fe-config\") pod \"kube-apiserver-proxy-ip-10-0-141-203.ec2.internal\" (UID: \"356e6d0e35ed5731a5b22ac92b6369fe\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" Jul 23 11:35:52.710414 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:52.710334 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:52.811216 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:52.811177 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:52.886830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:52.886789 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:52.890218 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:52.890194 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" Jul 23 11:35:52.912329 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:52.912279 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:53.012928 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:53.012829 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:53.092144 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.092113 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:53.113743 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:53.113711 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:53.212259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.212226 2554 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 23 11:35:53.212778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.212394 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:35:53.212778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.212427 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:35:53.212778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.212429 2554 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 23 11:35:53.214388 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:53.214365 2554 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-203.ec2.internal\" not found" Jul 23 11:35:53.281925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.281820 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-22 11:30:52 +0000 UTC" deadline="2028-03-22 00:36:39.733547363 +0000 UTC" Jul 23 11:35:53.281925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.281863 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14581h0m46.451687119s" Jul 23 11:35:53.282250 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.282233 2554 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:53.373249 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.373206 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" Jul 23 11:35:53.389761 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.389731 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:35:53.392142 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.392119 2554 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" Jul 23 11:35:53.402937 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.402911 2554 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 23 11:35:53.521144 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:53.521108 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd870a017be816cae8b0075a63393449b.slice/crio-3c589cd3c5795ff3f59da128da0cadd27941d5fd5d4ad0a057dc0c7a26af4b66 WatchSource:0}: Error finding container 3c589cd3c5795ff3f59da128da0cadd27941d5fd5d4ad0a057dc0c7a26af4b66: Status 404 returned error can't find the container with id 3c589cd3c5795ff3f59da128da0cadd27941d5fd5d4ad0a057dc0c7a26af4b66 Jul 23 11:35:53.521474 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:53.521457 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356e6d0e35ed5731a5b22ac92b6369fe.slice/crio-34de9ca46637db387c018d201d5778b7fac005b3505257f6c8e97ad1d283e487 WatchSource:0}: Error finding container 34de9ca46637db387c018d201d5778b7fac005b3505257f6c8e97ad1d283e487: Status 404 returned error can't find the container with id 34de9ca46637db387c018d201d5778b7fac005b3505257f6c8e97ad1d283e487 Jul 23 11:35:53.527773 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:53.527751 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:35:54.250837 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.250790 2554 apiserver.go:52] "Watching apiserver" Jul 23 11:35:54.260975 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.260082 2554 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 23 11:35:54.260975 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.260544 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-525mm","openshift-image-registry/node-ca-2gqwm","openshift-multus/network-metrics-daemon-sj4mv","openshift-network-diagnostics/network-check-target-svsd4","openshift-network-operator/iptables-alerter-9fvv5","openshift-ovn-kubernetes/ovnkube-node-48jpn","kube-system/global-pull-secret-syncer-8vr4l","kube-system/konnectivity-agent-dt86d","kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6","openshift-dns/node-resolver-dx4jr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal","openshift-multus/multus-additional-cni-plugins-4rmgk","openshift-multus/multus-bw5xj"] Jul 23 11:35:54.263437 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.263405 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.263575 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.263504 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:35:54.265844 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.265815 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.270082 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.269953 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 23 11:35:54.270082 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.269978 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.270256 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.270137 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.270256 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.270184 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 23 11:35:54.270349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.270316 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 23 11:35:54.270349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.270345 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nzlgs\"" Jul 23 11:35:54.270644 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.270548 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.270729 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.270647 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:35:54.272730 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.272312 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:54.272730 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.272381 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:35:54.273866 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.273840 2554 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 23 11:35:54.275154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.275131 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.277298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.277170 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.278426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.277804 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.278426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.277942 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.278426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.278038 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-9jgrc\"" Jul 23 11:35:54.278426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.278206 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 23 11:35:54.279729 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.279581 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.281670 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.281486 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 23 11:35:54.281670 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.281663 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.281823 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.281792 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 23 11:35:54.281952 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.281937 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 23 11:35:54.282069 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.282054 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-l49kp\"" Jul 23 11:35:54.282174 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.282161 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.282377 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.282360 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 23 11:35:54.282890 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.282870 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.284393 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.284370 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 23 11:35:54.284880 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.284860 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-xj877\"" Jul 23 11:35:54.285862 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.285840 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-twz25\"" Jul 23 11:35:54.286139 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286104 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.286219 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286154 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cnibin\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.286277 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-os-release\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.286333 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286274 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.286333 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286286 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b147eb79-42fd-4c49-a7a1-972af611f9db-kubelet-config\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.286435 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286338 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b147eb79-42fd-4c49-a7a1-972af611f9db-dbus\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.286435 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286368 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.286435 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286412 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjcbw\" (UniqueName: \"kubernetes.io/projected/50dbd128-630f-4bc1-b386-20cb0d159b8d-kube-api-access-fjcbw\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.286587 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286440 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:54.286587 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286478 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.286587 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286509 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.286587 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286580 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37f9f08f-1337-4856-a5e9-79ea5855395c-iptables-alerter-script\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.286820 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286645 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37f9f08f-1337-4856-a5e9-79ea5855395c-host-slash\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.286820 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286716 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58xmp\" (UniqueName: \"kubernetes.io/projected/37f9f08f-1337-4856-a5e9-79ea5855395c-kube-api-access-58xmp\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.286820 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286749 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-system-cni-dir\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.286820 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286816 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.287027 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286846 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.287027 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.286892 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hg2k\" (UniqueName: \"kubernetes.io/projected/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-kube-api-access-8hg2k\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.288779 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.288756 2554 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:54.288980 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.288961 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.289069 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.289051 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.289136 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.289070 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 23 11:35:54.289187 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.289009 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.289187 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.288973 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.289397 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.289376 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.289465 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.289449 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dsntn\"" Jul 23 11:35:54.292236 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.292214 2554 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 23 11:35:54.292326 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.292314 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 23 11:35:54.292481 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.292459 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gzxvp\"" Jul 23 11:35:54.292581 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.292518 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.292749 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.292732 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.294818 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.294151 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.296264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.296242 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 23 11:35:54.296506 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.296488 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-qbvb9\"" Jul 23 11:35:54.296841 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.296708 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 23 11:35:54.296841 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.296811 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.298969 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.298947 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 23 11:35:54.299106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.299089 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qcvpg\"" Jul 23 11:35:54.299211 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.299195 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 23 11:35:54.312112 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.312067 2554 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qzhvz" Jul 23 11:35:54.320388 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.320362 2554 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qzhvz" Jul 23 11:35:54.374937 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.374896 2554 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 23 11:35:54.387396 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387358 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovnkube-script-lib\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.387396 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387401 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.387662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387458 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwprt\" (UniqueName: \"kubernetes.io/projected/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-kube-api-access-vwprt\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.387662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387488 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-kubelet\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.387662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387507 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-run-netns\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.387662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387527 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-log-socket\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.387662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387572 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-cni-bin\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.387662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387649 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-var-lib-kubelet\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387679 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1057adf9-70f7-48f9-a433-c89f628aa741-serviceca\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387710 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387755 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-cnibin\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387781 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-multus-certs\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387805 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-slash\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387829 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysconfig\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.387850 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:54.387932 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387895 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysctl-conf\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.387953 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:35:54.887924811 +0000 UTC m=+2.993251203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.387981 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b147eb79-42fd-4c49-a7a1-972af611f9db-kubelet-config\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388015 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-socket-dir-parent\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388036 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b147eb79-42fd-4c49-a7a1-972af611f9db-kubelet-config\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388043 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388113 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37f9f08f-1337-4856-a5e9-79ea5855395c-iptables-alerter-script\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388154 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gz64\" (UniqueName: \"kubernetes.io/projected/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-kube-api-access-4gz64\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388215 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-systemd\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388257 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-var-lib-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388282 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gnwb\" (UniqueName: \"kubernetes.io/projected/5cd3fcea-7481-41e8-a0dc-3aad60336adf-kube-api-access-9gnwb\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.388302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388301 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-system-cni-dir\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388330 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-device-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388353 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388392 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-ovn\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388416 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-system-cni-dir\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388461 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysctl-d\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388522 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-sys\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388545 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-host\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388580 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klc5w\" (UniqueName: \"kubernetes.io/projected/c1bee8e2-b0db-4526-bd78-f7fece3659dd-kube-api-access-klc5w\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388603 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-cni-binary-copy\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388643 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-cni-bin\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388668 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-cni-multus\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388719 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/37f9f08f-1337-4856-a5e9-79ea5855395c-iptables-alerter-script\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388735 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a0cf057-5bc8-47b9-b9c6-76e11b9c5916-konnectivity-ca\") pod \"konnectivity-agent-dt86d\" (UID: \"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916\") " pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388775 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-env-overrides\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388797 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthnd\" (UniqueName: \"kubernetes.io/projected/1057adf9-70f7-48f9-a433-c89f628aa741-kube-api-access-sthnd\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.388830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388817 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b147eb79-42fd-4c49-a7a1-972af611f9db-dbus\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388851 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388894 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-os-release\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388941 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-hostroot\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388945 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b147eb79-42fd-4c49-a7a1-972af611f9db-dbus\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388963 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-conf-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.388999 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a0cf057-5bc8-47b9-b9c6-76e11b9c5916-agent-certs\") pod \"konnectivity-agent-dt86d\" (UID: \"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916\") " pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389020 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-etc-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389047 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58xmp\" (UniqueName: \"kubernetes.io/projected/37f9f08f-1337-4856-a5e9-79ea5855395c-kube-api-access-58xmp\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389093 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-daemon-config\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389132 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-node-log\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389166 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-kubernetes\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389222 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bee8e2-b0db-4526-bd78-f7fece3659dd-tmp\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389253 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1057adf9-70f7-48f9-a433-c89f628aa741-host\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389300 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd3fcea-7481-41e8-a0dc-3aad60336adf-hosts-file\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389361 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389407 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-system-cni-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.389516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389433 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-k8s-cni-cncf-io\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389458 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cnibin\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389488 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-os-release\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389514 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-cni-netd\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389539 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389566 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovnkube-config\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389593 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389639 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjcbw\" (UniqueName: \"kubernetes.io/projected/50dbd128-630f-4bc1-b386-20cb0d159b8d-kube-api-access-fjcbw\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovn-node-metrics-cert\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389696 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-lib-modules\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389725 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389755 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389784 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37f9f08f-1337-4856-a5e9-79ea5855395c-host-slash\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389821 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-kubelet\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389845 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd3fcea-7481-41e8-a0dc-3aad60336adf-tmp-dir\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389872 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.390310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389898 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hg2k\" (UniqueName: \"kubernetes.io/projected/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-kube-api-access-8hg2k\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389925 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-systemd-units\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389943 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx7v5\" (UniqueName: \"kubernetes.io/projected/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-kube-api-access-wx7v5\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389959 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.389984 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390004 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-run\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390032 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-tuned\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390049 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-cni-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390063 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-modprobe-d\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390087 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-systemd\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390111 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-netns\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390130 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-etc-kubernetes\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390153 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390250 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cnibin\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390350 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.390854 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-os-release\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.391108 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.390940 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:54.391824 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.391004 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:54.890983344 +0000 UTC m=+2.996309737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:54.391824 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.391004 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.391824 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.391048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/37f9f08f-1337-4856-a5e9-79ea5855395c-host-slash\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.391824 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.391155 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-cni-binary-copy\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.399850 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.399361 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:54.399850 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.399399 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:54.399850 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.399414 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:54.399850 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.399493 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:54.899472451 +0000 UTC m=+3.004798858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:54.401933 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.401897 2554 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 23 11:35:54.406147 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.406114 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hg2k\" (UniqueName: \"kubernetes.io/projected/b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b-kube-api-access-8hg2k\") pod \"multus-additional-cni-plugins-4rmgk\" (UID: \"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b\") " pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.406453 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.406423 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58xmp\" (UniqueName: \"kubernetes.io/projected/37f9f08f-1337-4856-a5e9-79ea5855395c-kube-api-access-58xmp\") pod \"iptables-alerter-9fvv5\" (UID: \"37f9f08f-1337-4856-a5e9-79ea5855395c\") " pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.406731 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.406711 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjcbw\" (UniqueName: \"kubernetes.io/projected/50dbd128-630f-4bc1-b386-20cb0d159b8d-kube-api-access-fjcbw\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.451710 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.451653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" event={"ID":"356e6d0e35ed5731a5b22ac92b6369fe","Type":"ContainerStarted","Data":"34de9ca46637db387c018d201d5778b7fac005b3505257f6c8e97ad1d283e487"} Jul 23 11:35:54.452944 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.452913 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" event={"ID":"d870a017be816cae8b0075a63393449b","Type":"ContainerStarted","Data":"3c589cd3c5795ff3f59da128da0cadd27941d5fd5d4ad0a057dc0c7a26af4b66"} Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.490910 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gz64\" (UniqueName: \"kubernetes.io/projected/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-kube-api-access-4gz64\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.490945 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-systemd\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.490969 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-var-lib-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.490995 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gnwb\" (UniqueName: \"kubernetes.io/projected/5cd3fcea-7481-41e8-a0dc-3aad60336adf-kube-api-access-9gnwb\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491017 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-device-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491038 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491061 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-ovn\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491083 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysctl-d\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491091 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-var-lib-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491108 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-sys\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491132 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-host\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491153 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klc5w\" (UniqueName: \"kubernetes.io/projected/c1bee8e2-b0db-4526-bd78-f7fece3659dd-kube-api-access-klc5w\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491163 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-systemd\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491176 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-cni-binary-copy\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491200 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-cni-bin\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491222 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-cni-multus\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.491835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491235 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-device-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491244 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491269 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a0cf057-5bc8-47b9-b9c6-76e11b9c5916-konnectivity-ca\") pod \"konnectivity-agent-dt86d\" (UID: \"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916\") " pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491283 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-ovn\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491292 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-env-overrides\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthnd\" (UniqueName: \"kubernetes.io/projected/1057adf9-70f7-48f9-a433-c89f628aa741-kube-api-access-sthnd\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491327 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-cni-multus\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491352 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-os-release\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491371 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-cni-bin\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491375 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-hostroot\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491401 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysctl-d\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491410 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-hostroot\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491414 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-conf-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491442 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a0cf057-5bc8-47b9-b9c6-76e11b9c5916-agent-certs\") pod \"konnectivity-agent-dt86d\" (UID: \"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916\") " pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491461 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-sys-fs\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491466 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-etc-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491462 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-sys\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-daemon-config\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491501 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-host\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491517 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-node-log\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491541 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-kubernetes\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491564 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bee8e2-b0db-4526-bd78-f7fece3659dd-tmp\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491588 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1057adf9-70f7-48f9-a433-c89f628aa741-host\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491611 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd3fcea-7481-41e8-a0dc-3aad60336adf-hosts-file\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491657 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-system-cni-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491686 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-k8s-cni-cncf-io\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491714 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-cni-netd\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491738 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491765 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovnkube-config\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491805 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovn-node-metrics-cert\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491827 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-lib-modules\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491866 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-kubelet\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491891 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd3fcea-7481-41e8-a0dc-3aad60336adf-tmp-dir\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-systemd-units\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491948 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx7v5\" (UniqueName: \"kubernetes.io/projected/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-kube-api-access-wx7v5\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.492946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491972 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.491995 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492022 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-run\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492045 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-tuned\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492046 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2a0cf057-5bc8-47b9-b9c6-76e11b9c5916-konnectivity-ca\") pod \"konnectivity-agent-dt86d\" (UID: \"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916\") " pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492071 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-cni-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492096 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-modprobe-d\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492127 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-systemd\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492149 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-cni-binary-copy\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492154 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-netns\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492201 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-conf-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492203 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-etc-kubernetes\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492234 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492262 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovnkube-script-lib\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492288 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492316 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwprt\" (UniqueName: \"kubernetes.io/projected/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-kube-api-access-vwprt\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.493405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.492501 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-env-overrides\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.494821 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-lib-modules\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.495681 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-var-lib-kubelet\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.495976 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cd3fcea-7481-41e8-a0dc-3aad60336adf-tmp-dir\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.497430 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-tuned\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.497639 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2a0cf057-5bc8-47b9-b9c6-76e11b9c5916-agent-certs\") pod \"konnectivity-agent-dt86d\" (UID: \"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916\") " pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.494711 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-os-release\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499207 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-kubelet\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499249 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-run-netns\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499249 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-etc-kubernetes\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499277 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-log-socket\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499295 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-cni-bin\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499313 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-var-lib-kubelet\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499320 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-etc-selinux\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499332 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1057adf9-70f7-48f9-a433-c89f628aa741-serviceca\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499343 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-etc-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499378 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-systemd-units\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499376 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-cnibin\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.499706 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499422 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-cnibin\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499422 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-multus-certs\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499452 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-multus-certs\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-run-openvswitch\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-slash\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-cni-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499505 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysconfig\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499518 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-kubernetes\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499540 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysctl-conf\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499542 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysconfig\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499557 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-slash\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499599 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-node-log\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499602 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-registration-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499638 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-modprobe-d\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499657 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-cni-netd\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499663 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-kubelet\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499672 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-socket-dir-parent\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499697 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.500573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499708 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-system-cni-dir\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499712 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-systemd\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499745 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-netns\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499764 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-run-netns\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499795 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-log-socket\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499816 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-daemon-config\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499823 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-cni-bin\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.499868 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cd3fcea-7481-41e8-a0dc-3aad60336adf-hosts-file\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500018 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-var-lib-kubelet\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500097 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/1057adf9-70f7-48f9-a433-c89f628aa741-host\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500162 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-run-ovn-kubernetes\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500215 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-multus-socket-dir-parent\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500218 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovnkube-script-lib\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500230 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-etc-sysctl-conf\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500256 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-host-run-k8s-cni-cncf-io\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500276 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500306 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500328 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-socket-dir\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.501298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500354 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1bee8e2-b0db-4526-bd78-f7fece3659dd-run\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.501940 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500383 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/1057adf9-70f7-48f9-a433-c89f628aa741-serviceca\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.501940 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.500671 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovnkube-config\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.502230 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.502211 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1bee8e2-b0db-4526-bd78-f7fece3659dd-tmp\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.503219 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.503081 2554 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:54.503569 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.503550 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-ovn-node-metrics-cert\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.505700 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.505677 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gz64\" (UniqueName: \"kubernetes.io/projected/833d6b5b-d2e2-4e91-bfa5-f10bd3acb872-kube-api-access-4gz64\") pod \"multus-bw5xj\" (UID: \"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872\") " pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.513322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.513294 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthnd\" (UniqueName: \"kubernetes.io/projected/1057adf9-70f7-48f9-a433-c89f628aa741-kube-api-access-sthnd\") pod \"node-ca-2gqwm\" (UID: \"1057adf9-70f7-48f9-a433-c89f628aa741\") " pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.513463 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.513409 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gnwb\" (UniqueName: \"kubernetes.io/projected/5cd3fcea-7481-41e8-a0dc-3aad60336adf-kube-api-access-9gnwb\") pod \"node-resolver-dx4jr\" (UID: \"5cd3fcea-7481-41e8-a0dc-3aad60336adf\") " pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.513535 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.513482 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwprt\" (UniqueName: \"kubernetes.io/projected/f5fe12fb-4437-45dd-a01b-ca57c32b6cbb-kube-api-access-vwprt\") pod \"aws-ebs-csi-driver-node-m4cs6\" (UID: \"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.513995 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.513975 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klc5w\" (UniqueName: \"kubernetes.io/projected/c1bee8e2-b0db-4526-bd78-f7fece3659dd-kube-api-access-klc5w\") pod \"tuned-525mm\" (UID: \"c1bee8e2-b0db-4526-bd78-f7fece3659dd\") " pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.514440 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.514424 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx7v5\" (UniqueName: \"kubernetes.io/projected/5a80ae7d-717a-4eac-a611-c6eaf9a544cf-kube-api-access-wx7v5\") pod \"ovnkube-node-48jpn\" (UID: \"5a80ae7d-717a-4eac-a611-c6eaf9a544cf\") " pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.583879 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.583838 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" Jul 23 11:35:54.598604 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.598573 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9fvv5" Jul 23 11:35:54.609591 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.609547 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:35:54.615470 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.615443 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-bw5xj" Jul 23 11:35:54.623449 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.623416 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-525mm" Jul 23 11:35:54.633218 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.633190 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-2gqwm" Jul 23 11:35:54.642231 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.642203 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" Jul 23 11:35:54.648993 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.648960 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dx4jr" Jul 23 11:35:54.656777 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.656747 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:35:54.903158 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.903120 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:54.903312 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.903188 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:54.903312 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:54.903220 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:54.903312 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903278 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:54.903312 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903306 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:54.903482 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903320 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:54.903482 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903319 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:54.903482 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903336 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:54.903482 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903406 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:55.903372771 +0000 UTC m=+4.008699161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:54.903482 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903422 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:55.903415832 +0000 UTC m=+4.008742222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:54.903482 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:54.903432 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:35:55.903426834 +0000 UTC m=+4.008753223 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:55.324295 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.324210 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:30:54 +0000 UTC" deadline="2028-02-20 21:34:34.70901335 +0000 UTC" Jul 23 11:35:55.324295 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.324287 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13857h58m39.384733416s" Jul 23 11:35:55.446963 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.446923 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:55.447135 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.447039 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:35:55.468730 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.468684 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fe12fb_4437_45dd_a01b_ca57c32b6cbb.slice/crio-13c488cffd2a27a06dddd0cbfe19241e7b72cf9d5f69f7566da97446894af577 WatchSource:0}: Error finding container 13c488cffd2a27a06dddd0cbfe19241e7b72cf9d5f69f7566da97446894af577: Status 404 returned error can't find the container with id 13c488cffd2a27a06dddd0cbfe19241e7b72cf9d5f69f7566da97446894af577 Jul 23 11:35:55.472442 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.472360 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd3fcea_7481_41e8_a0dc_3aad60336adf.slice/crio-da219e3bb995423eb6c14e65f6dc0ad61fd4db8c93a258d21bf2a97e9ab29242 WatchSource:0}: Error finding container da219e3bb995423eb6c14e65f6dc0ad61fd4db8c93a258d21bf2a97e9ab29242: Status 404 returned error can't find the container with id da219e3bb995423eb6c14e65f6dc0ad61fd4db8c93a258d21bf2a97e9ab29242 Jul 23 11:35:55.477501 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.477426 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1057adf9_70f7_48f9_a433_c89f628aa741.slice/crio-704e24a07fe28a9d152bdf2b7ddc5c26f8bfd7e5123e73954ed8828f91771e56 WatchSource:0}: Error finding container 704e24a07fe28a9d152bdf2b7ddc5c26f8bfd7e5123e73954ed8828f91771e56: Status 404 returned error can't find the container with id 704e24a07fe28a9d152bdf2b7ddc5c26f8bfd7e5123e73954ed8828f91771e56 Jul 23 11:35:55.478443 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.477861 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1bee8e2_b0db_4526_bd78_f7fece3659dd.slice/crio-a3d31593a89819a5fff498a0c5f6e73b7e359077f94d21e47f79b9b8c81eef8e WatchSource:0}: Error finding container a3d31593a89819a5fff498a0c5f6e73b7e359077f94d21e47f79b9b8c81eef8e: Status 404 returned error can't find the container with id a3d31593a89819a5fff498a0c5f6e73b7e359077f94d21e47f79b9b8c81eef8e Jul 23 11:35:55.479712 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.479561 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2e356d0_8ad6_4d20_8bc6_f99ee8a8555b.slice/crio-172f1e20f5a90f43ef56f6b277473e6137b226948eb7e32daddb98cb288e3462 WatchSource:0}: Error finding container 172f1e20f5a90f43ef56f6b277473e6137b226948eb7e32daddb98cb288e3462: Status 404 returned error can't find the container with id 172f1e20f5a90f43ef56f6b277473e6137b226948eb7e32daddb98cb288e3462 Jul 23 11:35:55.485649 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.485512 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod833d6b5b_d2e2_4e91_bfa5_f10bd3acb872.slice/crio-0823ec1c9bc381144072889ae7f1654aced3fe848b38eb3a09b2bab9a8b571a3 WatchSource:0}: Error finding container 0823ec1c9bc381144072889ae7f1654aced3fe848b38eb3a09b2bab9a8b571a3: Status 404 returned error can't find the container with id 0823ec1c9bc381144072889ae7f1654aced3fe848b38eb3a09b2bab9a8b571a3 Jul 23 11:35:55.486539 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:35:55.486469 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a80ae7d_717a_4eac_a611_c6eaf9a544cf.slice/crio-364515b90a18128d08281d4797242aec83b660d619b5835aec7bf9f307cd33a8 WatchSource:0}: Error finding container 364515b90a18128d08281d4797242aec83b660d619b5835aec7bf9f307cd33a8: Status 404 returned error can't find the container with id 364515b90a18128d08281d4797242aec83b660d619b5835aec7bf9f307cd33a8 Jul 23 11:35:55.791369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.791131 2554 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 23 11:35:55.910936 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.910832 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:55.910936 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.910907 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:55.910952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911016 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911072 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911090 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911099 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:35:57.911078015 +0000 UTC m=+6.016404427 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911103 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911016 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:55.911157 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911156 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:35:57.911141318 +0000 UTC m=+6.016467708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:55.911451 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:55.911176 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:35:57.911167324 +0000 UTC m=+6.016493721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:56.325107 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.325057 2554 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-22 11:30:54 +0000 UTC" deadline="2028-03-20 04:04:13.840289544 +0000 UTC" Jul 23 11:35:56.325107 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.325101 2554 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14536h28m17.515193487s" Jul 23 11:35:56.449896 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.449749 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:56.449896 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:56.449890 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:35:56.450554 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.450344 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:56.450554 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:56.450466 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:35:56.484701 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.484642 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"364515b90a18128d08281d4797242aec83b660d619b5835aec7bf9f307cd33a8"} Jul 23 11:35:56.491774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.491738 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw5xj" event={"ID":"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872","Type":"ContainerStarted","Data":"0823ec1c9bc381144072889ae7f1654aced3fe848b38eb3a09b2bab9a8b571a3"} Jul 23 11:35:56.499332 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.499289 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9fvv5" event={"ID":"37f9f08f-1337-4856-a5e9-79ea5855395c","Type":"ContainerStarted","Data":"163e6ce4992194143a44ba1906e0864b6fbe2a7c14edcc84109a7ead664c938c"} Jul 23 11:35:56.514756 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.514713 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerStarted","Data":"172f1e20f5a90f43ef56f6b277473e6137b226948eb7e32daddb98cb288e3462"} Jul 23 11:35:56.523147 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.523043 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dt86d" event={"ID":"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916","Type":"ContainerStarted","Data":"5396eff447b3de628a3200de9ffb90132d228910b013abc4de5a438c59c03ffe"} Jul 23 11:35:56.529281 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.529236 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2gqwm" event={"ID":"1057adf9-70f7-48f9-a433-c89f628aa741","Type":"ContainerStarted","Data":"704e24a07fe28a9d152bdf2b7ddc5c26f8bfd7e5123e73954ed8828f91771e56"} Jul 23 11:35:56.536831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.536789 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dx4jr" event={"ID":"5cd3fcea-7481-41e8-a0dc-3aad60336adf","Type":"ContainerStarted","Data":"da219e3bb995423eb6c14e65f6dc0ad61fd4db8c93a258d21bf2a97e9ab29242"} Jul 23 11:35:56.554501 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.554367 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-525mm" event={"ID":"c1bee8e2-b0db-4526-bd78-f7fece3659dd","Type":"ContainerStarted","Data":"a3d31593a89819a5fff498a0c5f6e73b7e359077f94d21e47f79b9b8c81eef8e"} Jul 23 11:35:56.570912 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.570872 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" event={"ID":"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb","Type":"ContainerStarted","Data":"13c488cffd2a27a06dddd0cbfe19241e7b72cf9d5f69f7566da97446894af577"} Jul 23 11:35:56.585492 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.585404 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" event={"ID":"356e6d0e35ed5731a5b22ac92b6369fe","Type":"ContainerStarted","Data":"67f62de32f22c1e6583108444ec9dd43ebe68aaae6e1ca7583e382d952862edf"} Jul 23 11:35:56.599516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.599477 2554 generic.go:356] "Generic (PLEG): container finished" podID="d870a017be816cae8b0075a63393449b" containerID="e5e3be62c4abff5b89d569d30e2308daaf3d1ff2ad382b5a925dc303efdd20b5" exitCode=0 Jul 23 11:35:56.599714 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.599528 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" event={"ID":"d870a017be816cae8b0075a63393449b","Type":"ContainerDied","Data":"e5e3be62c4abff5b89d569d30e2308daaf3d1ff2ad382b5a925dc303efdd20b5"} Jul 23 11:35:56.604378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:56.604320 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-203.ec2.internal" podStartSLOduration=3.60430055 podStartE2EDuration="3.60430055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:56.603318122 +0000 UTC m=+4.708644535" watchObservedRunningTime="2026-07-23 11:35:56.60430055 +0000 UTC m=+4.709626965" Jul 23 11:35:57.447778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:57.447739 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:57.448279 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.447899 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:35:57.630498 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:57.630455 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" event={"ID":"d870a017be816cae8b0075a63393449b","Type":"ContainerStarted","Data":"abf7aa128020d9f59a090ae810f8bd9cba108ec198040c33de161b33436df219"} Jul 23 11:35:57.652963 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:57.651764 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-203.ec2.internal" podStartSLOduration=4.651728075 podStartE2EDuration="4.651728075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:35:57.651558029 +0000 UTC m=+5.756884443" watchObservedRunningTime="2026-07-23 11:35:57.651728075 +0000 UTC m=+5.757054507" Jul 23 11:35:57.929765 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:57.929726 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:57.929954 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:57.929801 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:57.929954 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:57.929853 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:57.930095 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930036 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:35:57.930095 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930059 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:35:57.930095 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930072 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:57.930235 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930136 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:01.93011667 +0000 UTC m=+10.035443080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:35:57.930562 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930542 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:57.930666 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930598 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:01.930583076 +0000 UTC m=+10.035909472 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:35:57.930740 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930670 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:57.930740 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:57.930705 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:36:01.930693042 +0000 UTC m=+10.036019437 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:35:58.448081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:58.448047 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:35:58.448527 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:58.448245 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:35:58.448662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:58.448061 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:35:58.448731 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:58.448705 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:35:59.447347 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:35:59.447311 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:35:59.447593 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:35:59.447446 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:00.448223 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:00.447449 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:00.448223 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:00.447601 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:00.448223 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:00.448072 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:00.448223 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:00.448174 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:01.447236 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:01.447201 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:01.447438 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.447337 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:01.966992 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:01.966949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:01.967027 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:01.967080 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967121 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967149 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967197 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.96717659 +0000 UTC m=+18.072502981 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967214 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.967206426 +0000 UTC m=+18.072532816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967226 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967245 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967258 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:01.967494 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:01.967320 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:09.967302342 +0000 UTC m=+18.072628732 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:02.447743 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:02.447708 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:02.447927 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:02.447853 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:02.448291 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:02.448267 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:02.448400 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:02.448372 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:03.448141 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:03.447774 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:03.448141 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:03.447920 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:04.448656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:04.447816 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:04.448656 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:04.447987 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:04.448656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:04.448501 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:04.448656 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:04.448591 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:05.447832 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:05.447794 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:05.448002 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:05.447929 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:06.447908 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:06.447878 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:06.448324 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:06.447878 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:06.448324 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:06.448004 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:06.448324 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:06.448095 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:07.447125 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:07.447088 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:07.447307 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:07.447200 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:08.447750 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:08.447712 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:08.448191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:08.447713 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:08.448191 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:08.447854 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:08.448191 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:08.447919 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:09.447020 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:09.446982 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:09.447318 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:09.447128 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:10.032828 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:10.032787 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:10.032847 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:10.032884 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.032952 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.032958 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.033025 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:26.033005538 +0000 UTC m=+34.138331942 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.033042 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:36:26.033032212 +0000 UTC m=+34.138358601 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.033057 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.033080 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.033093 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:10.033386 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.033137 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:26.033124896 +0000 UTC m=+34.138451286 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:10.447556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:10.447520 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:10.447778 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.447697 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:10.447778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:10.447746 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:10.447900 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:10.447874 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:11.447457 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:11.447419 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:11.447926 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:11.447538 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:12.448408 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:12.448371 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:12.448962 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:12.448491 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:12.448962 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:12.448553 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:12.448962 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:12.448698 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:13.447179 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:13.447139 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:13.447437 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:13.447297 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:14.446993 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.446966 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:14.447394 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.447005 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:14.447394 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:14.447105 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:14.447394 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:14.447232 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:14.672823 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.672495 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"65c793a7ffc3f75bde756e2602eea7565ae601e01b132d88aa0e85d06303a8c4"} Jul 23 11:36:14.674453 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.674425 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-bw5xj" event={"ID":"833d6b5b-d2e2-4e91-bfa5-f10bd3acb872","Type":"ContainerStarted","Data":"d6f231dff1d741cfea85221442c3923a88f66d3ccf74bd849f38c816a9711d69"} Jul 23 11:36:14.681889 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.681499 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerStarted","Data":"1b2493699fc171a547c27fa5f5168b238c2c10d1ed1420844b8e2331ab4863ee"} Jul 23 11:36:14.684746 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.684717 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dt86d" event={"ID":"2a0cf057-5bc8-47b9-b9c6-76e11b9c5916","Type":"ContainerStarted","Data":"4b6681d5838c42e601f9f9adcce5d94818613fef9faa92a399397bd015a919b7"} Jul 23 11:36:14.687303 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.687265 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-2gqwm" event={"ID":"1057adf9-70f7-48f9-a433-c89f628aa741","Type":"ContainerStarted","Data":"2d8653ff2d49f1fd43d954446f5bc307ac4d08d0a1622b1eba084fa771350b59"} Jul 23 11:36:14.688573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.688545 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dx4jr" event={"ID":"5cd3fcea-7481-41e8-a0dc-3aad60336adf","Type":"ContainerStarted","Data":"cd7d95b6ea235cd3a05dcd56a09c016026ca03c02f2d71144d0f445d32e6483b"} Jul 23 11:36:14.690739 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.690685 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-525mm" event={"ID":"c1bee8e2-b0db-4526-bd78-f7fece3659dd","Type":"ContainerStarted","Data":"3f1a60540b69732bcd24be4a6f139052ea46b179ebed02f45fcdd09804ffc769"} Jul 23 11:36:14.692378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.692321 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" event={"ID":"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb","Type":"ContainerStarted","Data":"661f505b290ea0d8146d034aea3b29dc5ff6409332bbecc79b9cc4a78e27720b"} Jul 23 11:36:14.717825 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.717772 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-bw5xj" podStartSLOduration=3.7414722400000002 podStartE2EDuration="22.717751966s" totalImagesPullingTime="18.976279726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:14.695914869 +0000 UTC m=+22.801241301" watchObservedRunningTime="2026-07-23 11:36:14.717751966 +0000 UTC m=+22.823078491" Jul 23 11:36:14.718003 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.717910 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-525mm" podStartSLOduration=3.750548883 podStartE2EDuration="22.71790097s" totalImagesPullingTime="18.967352087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:14.717391938 +0000 UTC m=+22.822718361" watchObservedRunningTime="2026-07-23 11:36:14.71790097 +0000 UTC m=+22.823227383" Jul 23 11:36:14.734169 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.733719 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dt86d" podStartSLOduration=8.126125764 podStartE2EDuration="22.733696386s" totalImagesPullingTime="14.607570622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:14.733286024 +0000 UTC m=+22.838612435" watchObservedRunningTime="2026-07-23 11:36:14.733696386 +0000 UTC m=+22.839022801" Jul 23 11:36:14.771201 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.771132 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-2gqwm" podStartSLOduration=3.822893777 podStartE2EDuration="22.771112423s" totalImagesPullingTime="18.948218646s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:14.771038848 +0000 UTC m=+22.876365260" watchObservedRunningTime="2026-07-23 11:36:14.771112423 +0000 UTC m=+22.876438838" Jul 23 11:36:14.787170 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.787121 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dx4jr" podStartSLOduration=3.833164614 podStartE2EDuration="22.787100559s" totalImagesPullingTime="18.953935945s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:14.786007416 +0000 UTC m=+22.891333827" watchObservedRunningTime="2026-07-23 11:36:14.787100559 +0000 UTC m=+22.892426970" Jul 23 11:36:14.977196 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.977172 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:36:14.977746 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:14.977729 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:36:15.447398 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.447365 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:15.448098 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:15.447473 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:15.698647 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.698590 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"3f4a169c434f19bda1aad8826d51755e429beeef3503096b56f1207d0a5ef07f"} Jul 23 11:36:15.698874 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.698711 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"a9bc759870b39d15b1c08cb68cd30e34338ccaaf113520d6803e159f2db7ff72"} Jul 23 11:36:15.698874 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.698766 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"d1c9f8368f7e8a511c8bd1e8f515f0ffb0a6a9740b24b1b3e1bb934f6f0d4218"} Jul 23 11:36:15.698874 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.698824 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"2a50b35b7c53d1ab24fb2fbb39101da842a1536a515981252f3c3974635bad13"} Jul 23 11:36:15.698874 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.698866 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"d1e83bc7a7a00a2cea988674cab927d2aeac971bd1cd0b5c3ae481ab04bd1c9a"} Jul 23 11:36:15.700063 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.700036 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b" containerID="1b2493699fc171a547c27fa5f5168b238c2c10d1ed1420844b8e2331ab4863ee" exitCode=0 Jul 23 11:36:15.700185 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:15.700136 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerDied","Data":"1b2493699fc171a547c27fa5f5168b238c2c10d1ed1420844b8e2331ab4863ee"} Jul 23 11:36:16.090951 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.090915 2554 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 23 11:36:16.345224 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.345096 2554 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-23T11:36:16.090939126Z","UUID":"6262dcc6-1c56-4b7b-9a9a-47fb89d55d97","Handler":null,"Name":"","Endpoint":""} Jul 23 11:36:16.348021 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.347725 2554 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 23 11:36:16.348021 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.347776 2554 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 23 11:36:16.447846 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.447808 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:16.448364 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.447815 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:16.448364 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:16.447957 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:16.448364 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:16.448074 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:16.704807 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.704671 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9fvv5" event={"ID":"37f9f08f-1337-4856-a5e9-79ea5855395c","Type":"ContainerStarted","Data":"d6df57e2b5f8f197556edd32da926cf7c6c868e3be73eaeeecdfb1e3e4777f6a"} Jul 23 11:36:16.707128 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.707106 2554 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 11:36:16.707128 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.707109 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" event={"ID":"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb","Type":"ContainerStarted","Data":"6e7ae454eda99efd5b772e347a32edbf9e8759ed75de1ef8322339c5ea5712d8"} Jul 23 11:36:16.725311 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:16.725255 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9fvv5" podStartSLOduration=5.782185444 podStartE2EDuration="24.725237558s" totalImagesPullingTime="18.943052114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:16.724285754 +0000 UTC m=+24.829612190" watchObservedRunningTime="2026-07-23 11:36:16.725237558 +0000 UTC m=+24.830563969" Jul 23 11:36:17.447565 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:17.447262 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:17.447565 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:17.447406 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:17.713809 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:17.713764 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"73918b5d4a6f3b44b5625f87a45b4747cb13bdbd021fb665efaa6c3bb044ba73"} Jul 23 11:36:17.716339 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:17.716294 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" event={"ID":"f5fe12fb-4437-45dd-a01b-ca57c32b6cbb","Type":"ContainerStarted","Data":"89964ab2166797bacccc970f23ad756fb959e1c9814878a5c238c37c0dd8ac02"} Jul 23 11:36:17.733964 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:17.733907 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-m4cs6" podStartSLOduration=4.329377989 podStartE2EDuration="25.733882726s" totalImagesPullingTime="21.404504737s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:17.73377903 +0000 UTC m=+25.839105444" watchObservedRunningTime="2026-07-23 11:36:17.733882726 +0000 UTC m=+25.839209140" Jul 23 11:36:18.447115 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:18.447076 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:18.447115 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:18.447111 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:18.447437 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:18.447218 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:18.447437 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:18.447326 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:19.391335 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:19.391299 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dx4jr_5cd3fcea-7481-41e8-a0dc-3aad60336adf/dns-node-resolver/0.log" Jul 23 11:36:19.446991 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:19.446956 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:19.447180 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:19.447088 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:20.163722 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.163523 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2gqwm_1057adf9-70f7-48f9-a433-c89f628aa741/node-ca/0.log" Jul 23 11:36:20.447807 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.447713 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:20.448773 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.447710 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:20.448773 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:20.447821 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:20.448773 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:20.447914 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:20.725912 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.725822 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b" containerID="b0ac4be517ccb2f1f19fe6c3e7748b9b48ce26d1583ca2f9b604b15956000fe0" exitCode=0 Jul 23 11:36:20.726081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.725923 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerDied","Data":"b0ac4be517ccb2f1f19fe6c3e7748b9b48ce26d1583ca2f9b604b15956000fe0"} Jul 23 11:36:20.729879 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.729840 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" event={"ID":"5a80ae7d-717a-4eac-a611-c6eaf9a544cf","Type":"ContainerStarted","Data":"44ccc4aff55d3ca4b869ed0bc025b962dc397c2ec26fc494c513056d23fe816d"} Jul 23 11:36:20.730103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.730084 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:36:20.730197 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.730133 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:36:20.730197 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.730180 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:36:20.746576 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.746550 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:36:20.746741 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.746673 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:36:20.799600 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:20.799547 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" podStartSLOduration=9.789456549 podStartE2EDuration="28.799530674s" totalImagesPullingTime="19.010074125s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:20.799313559 +0000 UTC m=+28.904639971" watchObservedRunningTime="2026-07-23 11:36:20.799530674 +0000 UTC m=+28.904857085" Jul 23 11:36:21.341118 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:21.340911 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:36:21.341365 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:21.341225 2554 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 23 11:36:21.341648 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:21.341605 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dt86d" Jul 23 11:36:21.447416 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:21.447368 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:21.447609 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:21.447518 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:22.079876 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.079793 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-svsd4"] Jul 23 11:36:22.080393 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.079936 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:22.080393 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:22.080014 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:22.083533 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.083507 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sj4mv"] Jul 23 11:36:22.083704 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.083641 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:22.083779 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:22.083740 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:22.084135 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.084114 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8vr4l"] Jul 23 11:36:22.084242 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.084214 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:22.084322 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:22.084301 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:22.737527 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.737491 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b" containerID="f850406faacdabb2cb123626d5acc528622492b43aa20a3d4b0735407231876c" exitCode=0 Jul 23 11:36:22.737743 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:22.737543 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerDied","Data":"f850406faacdabb2cb123626d5acc528622492b43aa20a3d4b0735407231876c"} Jul 23 11:36:23.448259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:23.447776 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:23.448259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:23.447820 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:23.448259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:23.447793 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:23.448259 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:23.447915 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:23.448259 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:23.448015 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:23.448259 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:23.448113 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:24.744469 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:24.744243 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b" containerID="b2628a50db4ac32df0de51ce0d6cdbf45a8c3949e5aa7f37731b372c6622dd00" exitCode=0 Jul 23 11:36:24.744469 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:24.744336 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerDied","Data":"b2628a50db4ac32df0de51ce0d6cdbf45a8c3949e5aa7f37731b372c6622dd00"} Jul 23 11:36:25.447824 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:25.447791 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:25.448032 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:25.447790 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:25.448032 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:25.447926 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:25.448148 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:25.448028 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:25.448148 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:25.447791 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:25.448222 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:25.448145 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:26.061090 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:26.061043 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:26.061128 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:26.061177 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061210 2554 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061253 2554 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061287 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs podName:50dbd128-630f-4bc1-b386-20cb0d159b8d nodeName:}" failed. No retries permitted until 2026-07-23 11:36:58.061266033 +0000 UTC m=+66.166592426 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs") pod "network-metrics-daemon-sj4mv" (UID: "50dbd128-630f-4bc1-b386-20cb0d159b8d") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061301 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061313 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret podName:b147eb79-42fd-4c49-a7a1-972af611f9db nodeName:}" failed. No retries permitted until 2026-07-23 11:36:58.061295308 +0000 UTC m=+66.166621704 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret") pod "global-pull-secret-syncer-8vr4l" (UID: "b147eb79-42fd-4c49-a7a1-972af611f9db") : object "kube-system"/"original-pull-secret" not registered Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061318 2554 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061330 2554 projected.go:196] Error preparing data for projected volume kube-api-access-lx952 for pod openshift-network-diagnostics/network-check-target-svsd4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:26.061546 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:26.061370 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952 podName:a7ae9a69-4a8b-4ce7-930b-88c55f860b37 nodeName:}" failed. No retries permitted until 2026-07-23 11:36:58.061359595 +0000 UTC m=+66.166685986 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lx952" (UniqueName: "kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952") pod "network-check-target-svsd4" (UID: "a7ae9a69-4a8b-4ce7-930b-88c55f860b37") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 23 11:36:27.447602 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:27.447560 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:27.447602 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:27.447583 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:27.448241 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:27.447570 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:27.448241 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:27.447735 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:27.448241 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:27.447835 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:27.448241 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:27.447925 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:29.447914 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:29.447869 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:29.447914 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:29.447915 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:29.448341 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:29.447981 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:29.448341 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:29.447999 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:29.448341 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:29.448057 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:29.448341 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:29.448119 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:29.757478 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:29.757397 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b" containerID="6d19510c40bc7114829680a6c58e8af52e9f0caf5a05a78fcd22e2418ea2c97c" exitCode=0 Jul 23 11:36:29.757478 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:29.757434 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerDied","Data":"6d19510c40bc7114829680a6c58e8af52e9f0caf5a05a78fcd22e2418ea2c97c"} Jul 23 11:36:30.762791 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:30.762757 2554 generic.go:356] "Generic (PLEG): container finished" podID="b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b" containerID="5514bc1d16278842c67c8a8521f79e37c6f5474fd47d9fedf6d2dd594944d7c9" exitCode=0 Jul 23 11:36:30.763387 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:30.762820 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerDied","Data":"5514bc1d16278842c67c8a8521f79e37c6f5474fd47d9fedf6d2dd594944d7c9"} Jul 23 11:36:31.447263 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:31.447228 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:31.447263 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:31.447264 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:31.447523 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:31.447264 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:31.447523 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:31.447347 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:31.447523 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:31.447405 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:31.447523 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:31.447505 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:31.768246 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:31.768165 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" event={"ID":"b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b","Type":"ContainerStarted","Data":"a0f2298b85402a8f403f3cd2504d4ec5c6ecefb500c1891fbe8523d7ae46b00a"} Jul 23 11:36:31.796804 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:31.796757 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4rmgk" podStartSLOduration=10.27023988 podStartE2EDuration="39.796737823s" totalImagesPullingTime="29.526497943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:31.795466645 +0000 UTC m=+39.900793057" watchObservedRunningTime="2026-07-23 11:36:31.796737823 +0000 UTC m=+39.902064234" Jul 23 11:36:33.447679 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:33.447651 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:33.448069 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:33.447738 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:33.448069 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:33.447741 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:33.448069 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:33.447754 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:33.448069 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:33.447824 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:33.448069 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:33.447884 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:35.447715 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:35.447682 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:35.447715 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:35.447715 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:35.448171 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:35.447783 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:35.448171 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:35.447827 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:35.448171 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:35.447895 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:35.448171 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:35.447962 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:37.447641 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:37.447585 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:37.447641 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:37.447611 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:37.448155 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:37.447680 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:37.448155 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:37.447759 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:37.448155 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:37.447849 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:37.448155 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:37.447943 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:39.447688 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:39.447612 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:39.448190 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:39.447693 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:39.448190 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:39.447756 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:39.448190 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:39.447767 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8vr4l" podUID="b147eb79-42fd-4c49-a7a1-972af611f9db" Jul 23 11:36:39.448190 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:39.447843 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sj4mv" podUID="50dbd128-630f-4bc1-b386-20cb0d159b8d" Jul 23 11:36:39.448190 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:36:39.447937 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-svsd4" podUID="a7ae9a69-4a8b-4ce7-930b-88c55f860b37" Jul 23 11:36:41.166052 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.166028 2554 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-203.ec2.internal" event="NodeReady" Jul 23 11:36:41.166461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.166144 2554 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 23 11:36:41.245057 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.244968 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7mnkg"] Jul 23 11:36:41.269611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.269566 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b2bcz"] Jul 23 11:36:41.269771 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.269692 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.277271 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.277249 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 23 11:36:41.277410 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.277255 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-67ffh\"" Jul 23 11:36:41.278397 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.278378 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 23 11:36:41.293738 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.293710 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mnkg"] Jul 23 11:36:41.293860 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.293750 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b2bcz"] Jul 23 11:36:41.293860 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.293827 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.299132 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.299102 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 23 11:36:41.299286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.299142 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 23 11:36:41.299286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.299164 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 23 11:36:41.299286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.299171 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jxxcm\"" Jul 23 11:36:41.299286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.299102 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 23 11:36:41.371084 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371048 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5wqgc"] Jul 23 11:36:41.371535 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371512 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7ef981-69c7-4308-8032-9f6c186a3159-data-volume\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.371578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371544 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9nt\" (UniqueName: \"kubernetes.io/projected/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-kube-api-access-bb9nt\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.371578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371564 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697bc\" (UniqueName: \"kubernetes.io/projected/9a7ef981-69c7-4308-8032-9f6c186a3159-kube-api-access-697bc\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.371682 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371584 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7ef981-69c7-4308-8032-9f6c186a3159-crio-socket\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.371717 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-config-volume\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.371717 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371706 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-metrics-tls\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.371819 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371722 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-tmp-dir\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.371819 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371771 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7ef981-69c7-4308-8032-9f6c186a3159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.371819 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.371796 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7ef981-69c7-4308-8032-9f6c186a3159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.397724 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.397698 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.404209 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.404182 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 23 11:36:41.404332 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.404229 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 23 11:36:41.405352 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.405330 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pb4w9\"" Jul 23 11:36:41.405672 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.405652 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5wqgc"] Jul 23 11:36:41.406328 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.406312 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 23 11:36:41.447555 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.447498 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:41.447555 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.447552 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:41.447837 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.447552 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:41.451524 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.451505 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:36:41.451676 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.451647 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xtz4b\"" Jul 23 11:36:41.452138 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.452116 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:36:41.452235 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.452121 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:36:41.452337 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.452322 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:36:41.452418 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.452405 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7flkj\"" Jul 23 11:36:41.472690 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472662 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7ef981-69c7-4308-8032-9f6c186a3159-crio-socket\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.472849 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472709 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-config-volume\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.472849 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472727 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-metrics-tls\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.472849 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472747 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-tmp-dir\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.472849 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472773 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcg8p\" (UniqueName: \"kubernetes.io/projected/5b1ff801-8f56-491e-a923-93f65cae3c85-kube-api-access-hcg8p\") pod \"ingress-canary-5wqgc\" (UID: \"5b1ff801-8f56-491e-a923-93f65cae3c85\") " pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.472849 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472835 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7ef981-69c7-4308-8032-9f6c186a3159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.473087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472866 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1ff801-8f56-491e-a923-93f65cae3c85-cert\") pod \"ingress-canary-5wqgc\" (UID: \"5b1ff801-8f56-491e-a923-93f65cae3c85\") " pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.473087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472911 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7ef981-69c7-4308-8032-9f6c186a3159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.473087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472924 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9a7ef981-69c7-4308-8032-9f6c186a3159-crio-socket\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.473087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472955 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7ef981-69c7-4308-8032-9f6c186a3159-data-volume\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.473087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.472977 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9nt\" (UniqueName: \"kubernetes.io/projected/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-kube-api-access-bb9nt\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.473087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.473007 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697bc\" (UniqueName: \"kubernetes.io/projected/9a7ef981-69c7-4308-8032-9f6c186a3159-kube-api-access-697bc\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.473359 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.473187 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-tmp-dir\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.473407 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.473380 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-config-volume\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.473456 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.473436 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9a7ef981-69c7-4308-8032-9f6c186a3159-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.473527 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.473506 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9a7ef981-69c7-4308-8032-9f6c186a3159-data-volume\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.476947 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.476923 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-metrics-tls\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.477067 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.477038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9a7ef981-69c7-4308-8032-9f6c186a3159-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.485357 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.485326 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-697bc\" (UniqueName: \"kubernetes.io/projected/9a7ef981-69c7-4308-8032-9f6c186a3159-kube-api-access-697bc\") pod \"insights-runtime-extractor-b2bcz\" (UID: \"9a7ef981-69c7-4308-8032-9f6c186a3159\") " pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.485920 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.485901 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9nt\" (UniqueName: \"kubernetes.io/projected/0bf927f4-2ca3-486a-8a10-f0609e6a8bcb-kube-api-access-bb9nt\") pod \"dns-default-7mnkg\" (UID: \"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb\") " pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.573346 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.573309 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1ff801-8f56-491e-a923-93f65cae3c85-cert\") pod \"ingress-canary-5wqgc\" (UID: \"5b1ff801-8f56-491e-a923-93f65cae3c85\") " pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.573491 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.573380 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcg8p\" (UniqueName: \"kubernetes.io/projected/5b1ff801-8f56-491e-a923-93f65cae3c85-kube-api-access-hcg8p\") pod \"ingress-canary-5wqgc\" (UID: \"5b1ff801-8f56-491e-a923-93f65cae3c85\") " pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.575948 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.575926 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/5b1ff801-8f56-491e-a923-93f65cae3c85-cert\") pod \"ingress-canary-5wqgc\" (UID: \"5b1ff801-8f56-491e-a923-93f65cae3c85\") " pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.578819 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.578798 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:41.582594 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.582570 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcg8p\" (UniqueName: \"kubernetes.io/projected/5b1ff801-8f56-491e-a923-93f65cae3c85-kube-api-access-hcg8p\") pod \"ingress-canary-5wqgc\" (UID: \"5b1ff801-8f56-491e-a923-93f65cae3c85\") " pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.602484 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.602454 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b2bcz" Jul 23 11:36:41.706670 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.706609 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5wqgc" Jul 23 11:36:41.818332 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.818302 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7mnkg"] Jul 23 11:36:41.820825 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.820795 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b2bcz"] Jul 23 11:36:41.822314 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:41.822291 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bf927f4_2ca3_486a_8a10_f0609e6a8bcb.slice/crio-820ec197a0d5fa95e62c57357c3c4d04040a4d1c36e33499d86942e61ca7685c WatchSource:0}: Error finding container 820ec197a0d5fa95e62c57357c3c4d04040a4d1c36e33499d86942e61ca7685c: Status 404 returned error can't find the container with id 820ec197a0d5fa95e62c57357c3c4d04040a4d1c36e33499d86942e61ca7685c Jul 23 11:36:41.824500 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:41.824442 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a7ef981_69c7_4308_8032_9f6c186a3159.slice/crio-00acd9571e558e19174ca867d93f68d6f99c3f9d18dc916ab96045c073d91f7a WatchSource:0}: Error finding container 00acd9571e558e19174ca867d93f68d6f99c3f9d18dc916ab96045c073d91f7a: Status 404 returned error can't find the container with id 00acd9571e558e19174ca867d93f68d6f99c3f9d18dc916ab96045c073d91f7a Jul 23 11:36:41.862341 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:41.862310 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5wqgc"] Jul 23 11:36:41.865548 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:41.865518 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b1ff801_8f56_491e_a923_93f65cae3c85.slice/crio-25ed5d3edacb5eb6ea277058ec08d2be266dedf600c3c75d9de1473ad829bda6 WatchSource:0}: Error finding container 25ed5d3edacb5eb6ea277058ec08d2be266dedf600c3c75d9de1473ad829bda6: Status 404 returned error can't find the container with id 25ed5d3edacb5eb6ea277058ec08d2be266dedf600c3c75d9de1473ad829bda6 Jul 23 11:36:42.797001 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:42.796952 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5wqgc" event={"ID":"5b1ff801-8f56-491e-a923-93f65cae3c85","Type":"ContainerStarted","Data":"25ed5d3edacb5eb6ea277058ec08d2be266dedf600c3c75d9de1473ad829bda6"} Jul 23 11:36:42.799508 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:42.799415 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2bcz" event={"ID":"9a7ef981-69c7-4308-8032-9f6c186a3159","Type":"ContainerStarted","Data":"00acd9571e558e19174ca867d93f68d6f99c3f9d18dc916ab96045c073d91f7a"} Jul 23 11:36:42.801777 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:42.801731 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mnkg" event={"ID":"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb","Type":"ContainerStarted","Data":"820ec197a0d5fa95e62c57357c3c4d04040a4d1c36e33499d86942e61ca7685c"} Jul 23 11:36:48.568576 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.568480 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-fb7fcbcf6-kf82q"] Jul 23 11:36:48.570651 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.570614 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.574928 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.574488 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ctvxt\"" Jul 23 11:36:48.574928 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.574797 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 23 11:36:48.576997 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.575493 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 23 11:36:48.576997 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.575682 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 23 11:36:48.576997 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.576182 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 23 11:36:48.576997 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.576536 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 23 11:36:48.576997 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.576800 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 23 11:36:48.577379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.577076 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 23 11:36:48.587336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.587310 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb7fcbcf6-kf82q"] Jul 23 11:36:48.632684 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.632647 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-service-ca\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.632861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.632735 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-serving-cert\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.632861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.632768 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-oauth-config\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.632861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.632784 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-oauth-serving-cert\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.632861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.632816 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-config\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.633068 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.632898 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdv2\" (UniqueName: \"kubernetes.io/projected/eb7d165e-e557-486c-9f64-e3f40fa0049d-kube-api-access-txdv2\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.681183 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.681099 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm"] Jul 23 11:36:48.683709 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.683687 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:48.686251 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.686231 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 23 11:36:48.686251 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.686236 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-xxmd2\"" Jul 23 11:36:48.703435 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.703403 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm"] Jul 23 11:36:48.734187 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734152 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-serving-cert\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.734187 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-t5msm\" (UID: \"6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:48.734423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734226 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-oauth-config\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.734423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734241 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-oauth-serving-cert\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.734423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734297 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-config\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.734423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734350 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdv2\" (UniqueName: \"kubernetes.io/projected/eb7d165e-e557-486c-9f64-e3f40fa0049d-kube-api-access-txdv2\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.734423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.734370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-service-ca\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.735106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.735082 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-service-ca\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.735106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.735101 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-oauth-serving-cert\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.735270 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.735156 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-config\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.736739 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.736718 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-serving-cert\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.736850 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.736832 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-oauth-config\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.743559 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.743539 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdv2\" (UniqueName: \"kubernetes.io/projected/eb7d165e-e557-486c-9f64-e3f40fa0049d-kube-api-access-txdv2\") pod \"console-fb7fcbcf6-kf82q\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.819949 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.819905 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5wqgc" event={"ID":"5b1ff801-8f56-491e-a923-93f65cae3c85","Type":"ContainerStarted","Data":"a6ce2c9710d4ff88993957d627ad2d94aaf1eac342bc3cbdc32da8464a3f0e57"} Jul 23 11:36:48.821378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.821339 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2bcz" event={"ID":"9a7ef981-69c7-4308-8032-9f6c186a3159","Type":"ContainerStarted","Data":"2bbbcb162b7b463db3e8fbafdf39ee5e9574b12d974d6631efeaa939a21b61fb"} Jul 23 11:36:48.823002 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.822978 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mnkg" event={"ID":"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb","Type":"ContainerStarted","Data":"d1a1a9bd27474af96c38fb8e08dee8e194345a4182de1cf08cd6b15a23734139"} Jul 23 11:36:48.823117 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.823021 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7mnkg" event={"ID":"0bf927f4-2ca3-486a-8a10-f0609e6a8bcb","Type":"ContainerStarted","Data":"eb8f640d99decc20a81e210921deaba3b5aa84c73cd8ea9aff1159de4fb5c911"} Jul 23 11:36:48.823117 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.823062 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:48.835245 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.835214 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-t5msm\" (UID: \"6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:48.837749 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.837722 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-t5msm\" (UID: \"6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:48.841352 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.841305 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5wqgc" podStartSLOduration=1.325328278 podStartE2EDuration="7.841288952s" totalImagesPullingTime="6.515960674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.840598609 +0000 UTC m=+56.945925021" watchObservedRunningTime="2026-07-23 11:36:48.841288952 +0000 UTC m=+56.946615341" Jul 23 11:36:48.859978 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.859921 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7mnkg" podStartSLOduration=1.3011681130000001 podStartE2EDuration="7.859902617s" totalImagesPullingTime="6.558734504s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:48.859381531 +0000 UTC m=+56.964707944" watchObservedRunningTime="2026-07-23 11:36:48.859902617 +0000 UTC m=+56.965229007" Jul 23 11:36:48.885726 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.885687 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:48.992768 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:48.992722 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:49.071634 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:49.071587 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb7fcbcf6-kf82q"] Jul 23 11:36:49.074374 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:49.074346 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb7d165e_e557_486c_9f64_e3f40fa0049d.slice/crio-8d8baec3476c0bfd3ed05fedeb6fb291e226c169a502c65b393f2cd2cc3ce641 WatchSource:0}: Error finding container 8d8baec3476c0bfd3ed05fedeb6fb291e226c169a502c65b393f2cd2cc3ce641: Status 404 returned error can't find the container with id 8d8baec3476c0bfd3ed05fedeb6fb291e226c169a502c65b393f2cd2cc3ce641 Jul 23 11:36:49.164931 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:49.164893 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm"] Jul 23 11:36:49.169308 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:49.169270 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecd05d1_08ec_4ac0_b9a8_d50ee7b3ec96.slice/crio-1cfbffb7e6259c8e6e2537155a1fee28188c92170a0504f07707291af7001669 WatchSource:0}: Error finding container 1cfbffb7e6259c8e6e2537155a1fee28188c92170a0504f07707291af7001669: Status 404 returned error can't find the container with id 1cfbffb7e6259c8e6e2537155a1fee28188c92170a0504f07707291af7001669 Jul 23 11:36:49.828483 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:49.828382 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" event={"ID":"6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96","Type":"ContainerStarted","Data":"1cfbffb7e6259c8e6e2537155a1fee28188c92170a0504f07707291af7001669"} Jul 23 11:36:49.831488 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:49.831003 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb7fcbcf6-kf82q" event={"ID":"eb7d165e-e557-486c-9f64-e3f40fa0049d","Type":"ContainerStarted","Data":"8d8baec3476c0bfd3ed05fedeb6fb291e226c169a502c65b393f2cd2cc3ce641"} Jul 23 11:36:52.748881 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:52.748847 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-48jpn" Jul 23 11:36:53.844985 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.844940 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2bcz" event={"ID":"9a7ef981-69c7-4308-8032-9f6c186a3159","Type":"ContainerStarted","Data":"65058ab039191e792aec7beecf5137ea4a32a67860214cc29e0b86b98ecc8e53"} Jul 23 11:36:53.846797 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.846769 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb7fcbcf6-kf82q" event={"ID":"eb7d165e-e557-486c-9f64-e3f40fa0049d","Type":"ContainerStarted","Data":"df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6"} Jul 23 11:36:53.848407 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.848382 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" event={"ID":"6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96","Type":"ContainerStarted","Data":"d08f8fe079a07d1325c244a3521fffe986475bb7d7b371d0b4e60ab3aeee7cda"} Jul 23 11:36:53.848662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.848617 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:53.853366 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.853335 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" Jul 23 11:36:53.868844 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.868798 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-fb7fcbcf6-kf82q" podStartSLOduration=1.5254713039999999 podStartE2EDuration="5.868780496s" totalImagesPullingTime="4.343309192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:53.867552975 +0000 UTC m=+61.972879386" watchObservedRunningTime="2026-07-23 11:36:53.868780496 +0000 UTC m=+61.974106965" Jul 23 11:36:53.887356 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:53.887311 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-t5msm" podStartSLOduration=1.644796411 podStartE2EDuration="5.887295066s" totalImagesPullingTime="4.242498655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:53.886690083 +0000 UTC m=+61.992016494" watchObservedRunningTime="2026-07-23 11:36:53.887295066 +0000 UTC m=+61.992621477" Jul 23 11:36:54.760869 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.760832 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j"] Jul 23 11:36:54.779962 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.779936 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j"] Jul 23 11:36:54.780118 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.780081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.782794 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.782769 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 23 11:36:54.783772 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.783746 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 23 11:36:54.783892 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.783800 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 23 11:36:54.784039 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.784021 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 23 11:36:54.784204 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.783754 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dm97b\"" Jul 23 11:36:54.784415 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.784364 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 23 11:36:54.886052 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.886009 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.886524 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.886153 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrr5\" (UniqueName: \"kubernetes.io/projected/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-kube-api-access-qbrr5\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.886524 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.886279 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.886524 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.886424 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.986841 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.986787 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrr5\" (UniqueName: \"kubernetes.io/projected/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-kube-api-access-qbrr5\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.986841 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.986845 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.987157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.986949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.987157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.986992 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.987640 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.987603 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.989890 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.989852 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.990505 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.990479 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:54.997401 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:54.997327 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrr5\" (UniqueName: \"kubernetes.io/projected/c7d4d81b-1b5f-4855-8d2d-c6d88409fd72-kube-api-access-qbrr5\") pod \"prometheus-operator-5945b5cfd-ckr8j\" (UID: \"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:55.091179 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.091145 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" Jul 23 11:36:55.271251 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.271216 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j"] Jul 23 11:36:55.275540 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:55.275505 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d4d81b_1b5f_4855_8d2d_c6d88409fd72.slice/crio-7cc38940208a993a60a9442fc836aeb652269e1f8004a88be9a7ad4991d71018 WatchSource:0}: Error finding container 7cc38940208a993a60a9442fc836aeb652269e1f8004a88be9a7ad4991d71018: Status 404 returned error can't find the container with id 7cc38940208a993a60a9442fc836aeb652269e1f8004a88be9a7ad4991d71018 Jul 23 11:36:55.557124 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.557090 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7d7c5d85bb-kvm6v"] Jul 23 11:36:55.560194 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.560160 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.574111 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.574080 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 23 11:36:55.589238 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.589213 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7c5d85bb-kvm6v"] Jul 23 11:36:55.692121 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692080 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-oauth-config\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.692121 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692127 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-serving-cert\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.692424 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692224 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-oauth-serving-cert\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.692424 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692251 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llfpj\" (UniqueName: \"kubernetes.io/projected/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-kube-api-access-llfpj\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.692424 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692287 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-trusted-ca-bundle\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.692424 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692312 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-service-ca\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.692611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.692427 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-config\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.793867 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.793818 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-serving-cert\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794057 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.793968 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-oauth-serving-cert\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794057 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.794000 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llfpj\" (UniqueName: \"kubernetes.io/projected/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-kube-api-access-llfpj\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794057 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.794020 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-trusted-ca-bundle\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794057 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.794041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-service-ca\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794273 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.794101 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-config\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794273 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.794121 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-oauth-config\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.794971 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.794935 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-oauth-serving-cert\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.795098 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.795018 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-trusted-ca-bundle\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.795323 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.795301 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-config\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.795416 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.795350 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-service-ca\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.796789 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.796762 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-oauth-config\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.796903 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.796888 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-serving-cert\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.805063 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.805038 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llfpj\" (UniqueName: \"kubernetes.io/projected/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-kube-api-access-llfpj\") pod \"console-7d7c5d85bb-kvm6v\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:55.857215 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.857105 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" event={"ID":"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72","Type":"ContainerStarted","Data":"7cc38940208a993a60a9442fc836aeb652269e1f8004a88be9a7ad4991d71018"} Jul 23 11:36:55.870983 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:55.870937 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:36:57.131545 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:57.131521 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7d7c5d85bb-kvm6v"] Jul 23 11:36:57.134435 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:57.134409 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda95c22fa_bb3c_4a43_9a3e_21ce7c2ab5e3.slice/crio-4fee0a794305f04cd93152bd3a978c83ce19cbcf36925fb2d41a5e295f67c586 WatchSource:0}: Error finding container 4fee0a794305f04cd93152bd3a978c83ce19cbcf36925fb2d41a5e295f67c586: Status 404 returned error can't find the container with id 4fee0a794305f04cd93152bd3a978c83ce19cbcf36925fb2d41a5e295f67c586 Jul 23 11:36:57.866847 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:57.866810 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b2bcz" event={"ID":"9a7ef981-69c7-4308-8032-9f6c186a3159","Type":"ContainerStarted","Data":"1900df94247c1c0e935f2db42bad463cd1f0bedfd3528694409e49bd28dcee6c"} Jul 23 11:36:57.868372 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:57.868347 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c5d85bb-kvm6v" event={"ID":"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3","Type":"ContainerStarted","Data":"8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2"} Jul 23 11:36:57.868516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:57.868387 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c5d85bb-kvm6v" event={"ID":"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3","Type":"ContainerStarted","Data":"4fee0a794305f04cd93152bd3a978c83ce19cbcf36925fb2d41a5e295f67c586"} Jul 23 11:36:57.886236 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:57.886190 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b2bcz" podStartSLOduration=1.8228252710000001 podStartE2EDuration="16.886171313s" totalImagesPullingTime="15.063346042s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:36:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:57.885294752 +0000 UTC m=+65.990621165" watchObservedRunningTime="2026-07-23 11:36:57.886171313 +0000 UTC m=+65.991497728" Jul 23 11:36:57.906389 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:57.906198 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7d7c5d85bb-kvm6v" podStartSLOduration=2.90618306 podStartE2EDuration="2.90618306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:57.905243064 +0000 UTC m=+66.010569476" watchObservedRunningTime="2026-07-23 11:36:57.90618306 +0000 UTC m=+66.011509546" Jul 23 11:36:58.117556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.116891 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:58.117556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.116994 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:58.117556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.117032 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:58.119738 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.119710 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 23 11:36:58.120064 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.120034 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 23 11:36:58.120378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.119954 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 23 11:36:58.132072 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.132039 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 23 11:36:58.132423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.132164 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b147eb79-42fd-4c49-a7a1-972af611f9db-original-pull-secret\") pod \"global-pull-secret-syncer-8vr4l\" (UID: \"b147eb79-42fd-4c49-a7a1-972af611f9db\") " pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:58.132423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.132185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50dbd128-630f-4bc1-b386-20cb0d159b8d-metrics-certs\") pod \"network-metrics-daemon-sj4mv\" (UID: \"50dbd128-630f-4bc1-b386-20cb0d159b8d\") " pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:58.141210 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.141187 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx952\" (UniqueName: \"kubernetes.io/projected/a7ae9a69-4a8b-4ce7-930b-88c55f860b37-kube-api-access-lx952\") pod \"network-check-target-svsd4\" (UID: \"a7ae9a69-4a8b-4ce7-930b-88c55f860b37\") " pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:58.260000 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.259966 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xtz4b\"" Jul 23 11:36:58.263009 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.262989 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8vr4l" Jul 23 11:36:58.267859 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.267841 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:36:58.270336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.270318 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7flkj\"" Jul 23 11:36:58.279045 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.279021 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sj4mv" Jul 23 11:36:58.406095 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.406064 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8vr4l"] Jul 23 11:36:58.410316 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:58.410284 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb147eb79_42fd_4c49_a7a1_972af611f9db.slice/crio-46221e5c292febf68c7459c2b71ae101217eacac9bf59f6016f47264b463bacd WatchSource:0}: Error finding container 46221e5c292febf68c7459c2b71ae101217eacac9bf59f6016f47264b463bacd: Status 404 returned error can't find the container with id 46221e5c292febf68c7459c2b71ae101217eacac9bf59f6016f47264b463bacd Jul 23 11:36:58.422664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.422593 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-svsd4"] Jul 23 11:36:58.425698 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:58.425674 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ae9a69_4a8b_4ce7_930b_88c55f860b37.slice/crio-ad506873f8a9a9cca2a7301984e166033968282f0297617a10b6546ef89f7f31 WatchSource:0}: Error finding container ad506873f8a9a9cca2a7301984e166033968282f0297617a10b6546ef89f7f31: Status 404 returned error can't find the container with id ad506873f8a9a9cca2a7301984e166033968282f0297617a10b6546ef89f7f31 Jul 23 11:36:58.440176 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.440150 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sj4mv"] Jul 23 11:36:58.445142 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:36:58.445117 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50dbd128_630f_4bc1_b386_20cb0d159b8d.slice/crio-b4a3fb5afe0cbf4409fe220f0935eb961194d628bc52b355306e9eeec23365fc WatchSource:0}: Error finding container b4a3fb5afe0cbf4409fe220f0935eb961194d628bc52b355306e9eeec23365fc: Status 404 returned error can't find the container with id b4a3fb5afe0cbf4409fe220f0935eb961194d628bc52b355306e9eeec23365fc Jul 23 11:36:58.832913 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.832886 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7mnkg" Jul 23 11:36:58.875116 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.874828 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" event={"ID":"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72","Type":"ContainerStarted","Data":"c20e40ac6d91eb74fa33c23657416217c54152e93303d600fb63a799853c8668"} Jul 23 11:36:58.875116 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.874894 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" event={"ID":"c7d4d81b-1b5f-4855-8d2d-c6d88409fd72","Type":"ContainerStarted","Data":"20f5e0a0a7609dee4c2ef9326601fb7d7fd9fa1cbea3b6b45f4db98e2b8062ad"} Jul 23 11:36:58.876447 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.876414 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-svsd4" event={"ID":"a7ae9a69-4a8b-4ce7-930b-88c55f860b37","Type":"ContainerStarted","Data":"ad506873f8a9a9cca2a7301984e166033968282f0297617a10b6546ef89f7f31"} Jul 23 11:36:58.877698 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.877668 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8vr4l" event={"ID":"b147eb79-42fd-4c49-a7a1-972af611f9db","Type":"ContainerStarted","Data":"46221e5c292febf68c7459c2b71ae101217eacac9bf59f6016f47264b463bacd"} Jul 23 11:36:58.879087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.879065 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sj4mv" event={"ID":"50dbd128-630f-4bc1-b386-20cb0d159b8d","Type":"ContainerStarted","Data":"b4a3fb5afe0cbf4409fe220f0935eb961194d628bc52b355306e9eeec23365fc"} Jul 23 11:36:58.886707 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.886686 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:58.886777 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.886735 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:58.891546 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.891525 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:36:58.893665 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:58.893438 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-ckr8j" podStartSLOduration=2.295382633 podStartE2EDuration="4.893420523s" totalImagesPullingTime="2.59803789s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:36:58.89278013 +0000 UTC m=+66.998106554" watchObservedRunningTime="2026-07-23 11:36:58.893420523 +0000 UTC m=+66.998746935" Jul 23 11:36:59.907531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:59.907455 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sj4mv" event={"ID":"50dbd128-630f-4bc1-b386-20cb0d159b8d","Type":"ContainerStarted","Data":"cc4caa61c660f9f671bb2fb5af881fdcd6178ccbc98057e758887ba50fb152c8"} Jul 23 11:36:59.913866 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:36:59.913793 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:37:00.914441 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:00.913980 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sj4mv" event={"ID":"50dbd128-630f-4bc1-b386-20cb0d159b8d","Type":"ContainerStarted","Data":"7478f080e699fc726f3a42a11842e785f10adb020ba1858a91b8172f5a0def28"} Jul 23 11:37:01.283391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.283253 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sj4mv" podStartSLOduration=68.000391808 podStartE2EDuration="1m9.28322791s" totalImagesPullingTime="1.282836102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:00.93477323 +0000 UTC m=+69.040099653" watchObservedRunningTime="2026-07-23 11:37:01.28322791 +0000 UTC m=+69.388554325" Jul 23 11:37:01.284508 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.284482 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn"] Jul 23 11:37:01.302411 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.302380 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vwg66"] Jul 23 11:37:01.302553 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.302499 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.311921 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.311867 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-zjdfb\"" Jul 23 11:37:01.311921 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.311899 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 23 11:37:01.311921 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.311899 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 23 11:37:01.316447 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.316428 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn"] Jul 23 11:37:01.316565 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.316533 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.318768 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.318749 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r28nh\"" Jul 23 11:37:01.319186 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.319166 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 23 11:37:01.319318 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.319253 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 23 11:37:01.319391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.319382 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 23 11:37:01.329247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.329219 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-npmc7"] Jul 23 11:37:01.348566 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.348533 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vwg66"] Jul 23 11:37:01.348719 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.348676 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.352256 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.352230 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mhh7m\"" Jul 23 11:37:01.352527 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.352509 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 23 11:37:01.352835 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.352812 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 23 11:37:01.353013 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.352996 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 23 11:37:01.445183 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445144 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfgjk\" (UniqueName: \"kubernetes.io/projected/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-api-access-pfgjk\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.445391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445197 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.445391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445267 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbb9v\" (UniqueName: \"kubernetes.io/projected/3b9a134f-b88e-40ca-9749-9d15329fd19e-kube-api-access-cbb9v\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445303 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4474e3ea-0e2d-4f93-97ec-249c45cc9827-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.445391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445335 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-sys\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445365 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-tls\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445391 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445382 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445447 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445488 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445517 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445546 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5mp7\" (UniqueName: \"kubernetes.io/projected/4204af03-fa63-43c4-be7a-e7836115dccd-kube-api-access-w5mp7\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445572 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-accelerators-collector-config\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445605 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-wtmp\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445646 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4204af03-fa63-43c4-be7a-e7836115dccd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.445979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445708 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a134f-b88e-40ca-9749-9d15329fd19e-metrics-client-ca\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445734 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-textfile\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445759 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.445979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445792 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-root\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.445979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.445839 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4474e3ea-0e2d-4f93-97ec-249c45cc9827-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.546990 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.546939 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.546999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547025 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5mp7\" (UniqueName: \"kubernetes.io/projected/4204af03-fa63-43c4-be7a-e7836115dccd-kube-api-access-w5mp7\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547044 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-accelerators-collector-config\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547073 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-wtmp\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547094 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4204af03-fa63-43c4-be7a-e7836115dccd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:37:01.547113 2554 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547135 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a134f-b88e-40ca-9749-9d15329fd19e-metrics-client-ca\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547164 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547162 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-textfile\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:37:01.547188 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-tls podName:4204af03-fa63-43c4-be7a-e7836115dccd nodeName:}" failed. No retries permitted until 2026-07-23 11:37:02.04716852 +0000 UTC m=+70.152494927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-tls") pod "openshift-state-metrics-79f6958cdf-gkzsn" (UID: "4204af03-fa63-43c4-be7a-e7836115dccd") : secret "openshift-state-metrics-tls" not found Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547252 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547299 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-root\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547324 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4474e3ea-0e2d-4f93-97ec-249c45cc9827-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547397 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfgjk\" (UniqueName: \"kubernetes.io/projected/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-api-access-pfgjk\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547407 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-wtmp\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547507 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-root\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547550 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbb9v\" (UniqueName: \"kubernetes.io/projected/3b9a134f-b88e-40ca-9749-9d15329fd19e-kube-api-access-cbb9v\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.547611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547587 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4474e3ea-0e2d-4f93-97ec-249c45cc9827-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.548177 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547937 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-textfile\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.548177 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.547999 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4474e3ea-0e2d-4f93-97ec-249c45cc9827-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.548177 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548136 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-accelerators-collector-config\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.548342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548193 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-sys\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.548342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548233 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-tls\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.548342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548251 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3b9a134f-b88e-40ca-9749-9d15329fd19e-metrics-client-ca\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.548342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548279 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3b9a134f-b88e-40ca-9749-9d15329fd19e-sys\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.548342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548312 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.548342 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548313 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4474e3ea-0e2d-4f93-97ec-249c45cc9827-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.548665 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548350 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4204af03-fa63-43c4-be7a-e7836115dccd-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.548665 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548376 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.548665 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.548557 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.550182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.550158 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.550860 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.550830 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3b9a134f-b88e-40ca-9749-9d15329fd19e-node-exporter-tls\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.551286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.551254 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.551370 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.551322 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.551928 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.551906 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.559881 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.559862 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbb9v\" (UniqueName: \"kubernetes.io/projected/3b9a134f-b88e-40ca-9749-9d15329fd19e-kube-api-access-cbb9v\") pod \"node-exporter-npmc7\" (UID: \"3b9a134f-b88e-40ca-9749-9d15329fd19e\") " pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:01.559976 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.559897 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5mp7\" (UniqueName: \"kubernetes.io/projected/4204af03-fa63-43c4-be7a-e7836115dccd-kube-api-access-w5mp7\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:01.563303 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.563274 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfgjk\" (UniqueName: \"kubernetes.io/projected/4474e3ea-0e2d-4f93-97ec-249c45cc9827-kube-api-access-pfgjk\") pod \"kube-state-metrics-56c5594674-vwg66\" (UID: \"4474e3ea-0e2d-4f93-97ec-249c45cc9827\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.626551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.626511 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" Jul 23 11:37:01.659686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:01.659658 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-npmc7" Jul 23 11:37:02.052526 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.052465 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:02.055384 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.055359 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4204af03-fa63-43c4-be7a-e7836115dccd-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-gkzsn\" (UID: \"4204af03-fa63-43c4-be7a-e7836115dccd\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:02.214685 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.214617 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" Jul 23 11:37:02.229725 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.229690 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:37:02.246747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.246712 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.249947 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.249926 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 11:37:02.250686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.250457 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 11:37:02.250686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.250502 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 11:37:02.250686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.250519 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 11:37:02.250686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.250546 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wl6r9\"" Jul 23 11:37:02.251181 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.251159 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 11:37:02.251181 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.251176 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 11:37:02.251345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.251198 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 11:37:02.251345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.251210 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 11:37:02.251345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.251180 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 11:37:02.261133 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.260511 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:37:02.354496 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354401 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hghdh\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-kube-api-access-hghdh\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354496 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354448 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354496 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354482 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354534 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354582 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354677 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354712 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354733 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.354831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354791 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.355106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354856 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.355106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354890 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-out\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.355106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354918 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-web-config\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.355106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.354995 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455317 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hghdh\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-kube-api-access-hghdh\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455360 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455384 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455516 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455411 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455705 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455525 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455705 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455570 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455705 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455591 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455705 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:37:02.455612 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle podName:5dd808d4-1bf0-4402-a28e-ed17d13613d4 nodeName:}" failed. No retries permitted until 2026-07-23 11:37:02.955589248 +0000 UTC m=+71.060915642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4") : configmap references non-existent config key: ca-bundle.crt Jul 23 11:37:02.455705 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455672 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455958 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455719 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455958 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455774 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455958 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455811 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-out\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455958 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455839 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-web-config\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.455958 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.455869 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.456191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.456056 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.457353 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.457213 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.460930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.460908 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-out\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.461089 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.460974 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.461208 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.461129 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.461578 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.461551 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.461703 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.461679 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-volume\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.462215 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.462193 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.464379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.464354 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hghdh\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-kube-api-access-hghdh\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.466408 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.466262 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.467720 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.467092 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.467823 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.467803 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-web-config\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.960034 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.960000 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:02.960870 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:02.960842 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:03.159247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.159211 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:37:03.294281 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:37:03.294189 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b9a134f_b88e_40ca_9749_9d15329fd19e.slice/crio-5035436d250ec595cb8fd701a32d336667baa16823b40f63aa77cca5680e9ee1 WatchSource:0}: Error finding container 5035436d250ec595cb8fd701a32d336667baa16823b40f63aa77cca5680e9ee1: Status 404 returned error can't find the container with id 5035436d250ec595cb8fd701a32d336667baa16823b40f63aa77cca5680e9ee1 Jul 23 11:37:03.464422 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.464392 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn"] Jul 23 11:37:03.468947 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:37:03.468914 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4204af03_fa63_43c4_be7a_e7836115dccd.slice/crio-7f41144728a33828c905065342982d6fd8b71c97109d45b7e7e1837d78bd8530 WatchSource:0}: Error finding container 7f41144728a33828c905065342982d6fd8b71c97109d45b7e7e1837d78bd8530: Status 404 returned error can't find the container with id 7f41144728a33828c905065342982d6fd8b71c97109d45b7e7e1837d78bd8530 Jul 23 11:37:03.692840 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.692810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-vwg66"] Jul 23 11:37:03.696411 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:37:03.696378 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4474e3ea_0e2d_4f93_97ec_249c45cc9827.slice/crio-f5d2befac1a7276a9305c8fc48c624b6da3653c0c247f3a199819c2ef4d4bb9d WatchSource:0}: Error finding container f5d2befac1a7276a9305c8fc48c624b6da3653c0c247f3a199819c2ef4d4bb9d: Status 404 returned error can't find the container with id f5d2befac1a7276a9305c8fc48c624b6da3653c0c247f3a199819c2ef4d4bb9d Jul 23 11:37:03.702073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.702050 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:37:03.708971 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:37:03.708943 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd808d4_1bf0_4402_a28e_ed17d13613d4.slice/crio-741e0cad216aff516fee24b7a98358ce6c20dfa5af0be7f6d9b334840631d398 WatchSource:0}: Error finding container 741e0cad216aff516fee24b7a98358ce6c20dfa5af0be7f6d9b334840631d398: Status 404 returned error can't find the container with id 741e0cad216aff516fee24b7a98358ce6c20dfa5af0be7f6d9b334840631d398 Jul 23 11:37:03.929031 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.928482 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-svsd4" event={"ID":"a7ae9a69-4a8b-4ce7-930b-88c55f860b37","Type":"ContainerStarted","Data":"40fbf6f01830f400113ff954637715f2a751dc38e7a164765bdd8895cf5b0739"} Jul 23 11:37:03.929031 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.928574 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:37:03.930238 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.930211 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8vr4l" event={"ID":"b147eb79-42fd-4c49-a7a1-972af611f9db","Type":"ContainerStarted","Data":"d858f67585b43861b725090e9be249def12bafc3e252c95f9710d065d2dc4408"} Jul 23 11:37:03.931701 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.931670 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"741e0cad216aff516fee24b7a98358ce6c20dfa5af0be7f6d9b334840631d398"} Jul 23 11:37:03.933672 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.933648 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" event={"ID":"4474e3ea-0e2d-4f93-97ec-249c45cc9827","Type":"ContainerStarted","Data":"f5d2befac1a7276a9305c8fc48c624b6da3653c0c247f3a199819c2ef4d4bb9d"} Jul 23 11:37:03.935193 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.935163 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npmc7" event={"ID":"3b9a134f-b88e-40ca-9749-9d15329fd19e","Type":"ContainerStarted","Data":"5035436d250ec595cb8fd701a32d336667baa16823b40f63aa77cca5680e9ee1"} Jul 23 11:37:03.937584 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.937558 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" event={"ID":"4204af03-fa63-43c4-be7a-e7836115dccd","Type":"ContainerStarted","Data":"dec0f373a08fd40ada671e71cf0e345457cb9830f7640b21bf66db28bfc81c58"} Jul 23 11:37:03.937715 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.937606 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" event={"ID":"4204af03-fa63-43c4-be7a-e7836115dccd","Type":"ContainerStarted","Data":"31112309a2f9d8412ad6d27c3223c32b61af6a7b1ecab5ff6ade29b702b66439"} Jul 23 11:37:03.937715 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.937641 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" event={"ID":"4204af03-fa63-43c4-be7a-e7836115dccd","Type":"ContainerStarted","Data":"7f41144728a33828c905065342982d6fd8b71c97109d45b7e7e1837d78bd8530"} Jul 23 11:37:03.954217 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:03.954159 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-svsd4" podStartSLOduration=67.088819918 podStartE2EDuration="1m11.954139205s" totalImagesPullingTime="4.865319287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:03.953472106 +0000 UTC m=+72.058798518" watchObservedRunningTime="2026-07-23 11:37:03.954139205 +0000 UTC m=+72.059465616" Jul 23 11:37:04.943731 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:04.943698 2554 generic.go:356] "Generic (PLEG): container finished" podID="3b9a134f-b88e-40ca-9749-9d15329fd19e" containerID="07b58c44de7376caebbaab11bd18a8d5f9a02b3db870e2318a733c53e46d8b8b" exitCode=0 Jul 23 11:37:04.944158 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:04.943799 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npmc7" event={"ID":"3b9a134f-b88e-40ca-9749-9d15329fd19e","Type":"ContainerDied","Data":"07b58c44de7376caebbaab11bd18a8d5f9a02b3db870e2318a733c53e46d8b8b"} Jul 23 11:37:04.979712 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:04.979648 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8vr4l" podStartSLOduration=68.08356089 podStartE2EDuration="1m12.979601909s" totalImagesPullingTime="4.896041019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 11:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:03.97523924 +0000 UTC m=+72.080565681" watchObservedRunningTime="2026-07-23 11:37:04.979601909 +0000 UTC m=+73.084928321" Jul 23 11:37:05.871316 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.871278 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:37:05.871518 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.871370 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:37:05.876408 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.876380 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:37:05.951355 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.951323 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" event={"ID":"4474e3ea-0e2d-4f93-97ec-249c45cc9827","Type":"ContainerStarted","Data":"b90e3c61a2f8ce32944b37a0e1dc6a0fff00be7374d7e88e64f0cad54e184266"} Jul 23 11:37:05.951861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.951393 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" event={"ID":"4474e3ea-0e2d-4f93-97ec-249c45cc9827","Type":"ContainerStarted","Data":"8e2eb84ff9f621436955e4fa5adeeee698591bdf835712b4c6f59f090219f6ce"} Jul 23 11:37:05.951861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.951422 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" event={"ID":"4474e3ea-0e2d-4f93-97ec-249c45cc9827","Type":"ContainerStarted","Data":"7f8700477aff988a7e3171d9480f433f8398dee87c9aca763732ef1472f7173d"} Jul 23 11:37:05.954417 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.954384 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npmc7" event={"ID":"3b9a134f-b88e-40ca-9749-9d15329fd19e","Type":"ContainerStarted","Data":"8eea7c7023c62e9ad946495aa01b40521b1ef15fd1bc1f69517ebe0005a0d8c8"} Jul 23 11:37:05.954656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.954446 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-npmc7" event={"ID":"3b9a134f-b88e-40ca-9749-9d15329fd19e","Type":"ContainerStarted","Data":"c582aaccf7d7032062e4c492a0cc128c636424d17ae6fc4bd0c0a8cbcf853ab2"} Jul 23 11:37:05.957157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.957130 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" event={"ID":"4204af03-fa63-43c4-be7a-e7836115dccd","Type":"ContainerStarted","Data":"3097cbb354621e59cda38cb13228bdb30f98ca04ea2a7c3e0c085b8b6f0a603d"} Jul 23 11:37:05.959041 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.959013 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9" exitCode=0 Jul 23 11:37:05.959178 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.959100 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9"} Jul 23 11:37:05.963467 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.963442 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:37:05.984436 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:05.984385 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-vwg66" podStartSLOduration=3.144765978 podStartE2EDuration="4.984368551s" totalImagesPullingTime="1.839602573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:05.981338448 +0000 UTC m=+74.086664860" watchObservedRunningTime="2026-07-23 11:37:05.984368551 +0000 UTC m=+74.089694962" Jul 23 11:37:06.031212 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.031128 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d7c5d85bb-kvm6v"] Jul 23 11:37:06.061812 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.061772 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-d5b46f9c-jrgj5"] Jul 23 11:37:06.065300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.065265 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.091689 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.091014 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d5b46f9c-jrgj5"] Jul 23 11:37:06.100752 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.100559 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-gkzsn" podStartSLOduration=3.152470853 podStartE2EDuration="5.100537679s" totalImagesPullingTime="1.948066826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:06.098999775 +0000 UTC m=+74.204326209" watchObservedRunningTime="2026-07-23 11:37:06.100537679 +0000 UTC m=+74.205864091" Jul 23 11:37:06.130399 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.130349 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-npmc7" podStartSLOduration=4.127092726 podStartE2EDuration="5.130331511s" totalImagesPullingTime="924.385939ms" totalInitContainerRuntime="78.852846ms" isStatefulPod=true podCreationTimestamp="2026-07-23 11:37:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:06.129519917 +0000 UTC m=+74.234846340" watchObservedRunningTime="2026-07-23 11:37:06.130331511 +0000 UTC m=+74.235657922" Jul 23 11:37:06.190072 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190039 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-serving-cert\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.190072 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190072 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-oauth-serving-cert\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.190264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190092 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-oauth-config\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.190264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190127 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-console-config\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.190264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190151 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f288b\" (UniqueName: \"kubernetes.io/projected/35020295-870c-4c88-b2ab-12970419a2dd-kube-api-access-f288b\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.190264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190173 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-service-ca\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.190264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.190193 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-trusted-ca-bundle\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291647 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291582 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-console-config\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291647 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291649 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f288b\" (UniqueName: \"kubernetes.io/projected/35020295-870c-4c88-b2ab-12970419a2dd-kube-api-access-f288b\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291887 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291689 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-service-ca\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291887 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291717 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-trusted-ca-bundle\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291887 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291786 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-serving-cert\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291887 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291811 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-oauth-serving-cert\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.291887 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.291835 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-oauth-config\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.292486 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.292452 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-console-config\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.292748 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.292528 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-oauth-serving-cert\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.292748 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.292564 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-service-ca\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.292944 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.292774 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-trusted-ca-bundle\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.294408 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.294383 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-serving-cert\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.294547 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.294521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-oauth-config\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.300548 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.300527 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f288b\" (UniqueName: \"kubernetes.io/projected/35020295-870c-4c88-b2ab-12970419a2dd-kube-api-access-f288b\") pod \"console-d5b46f9c-jrgj5\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.375897 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.375857 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:06.518183 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.518152 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d5b46f9c-jrgj5"] Jul 23 11:37:06.522345 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:37:06.522313 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35020295_870c_4c88_b2ab_12970419a2dd.slice/crio-c53677117f41be77fcf8f55d7bdeb666c878064e6a2d5d71d991b647c75c1f67 WatchSource:0}: Error finding container c53677117f41be77fcf8f55d7bdeb666c878064e6a2d5d71d991b647c75c1f67: Status 404 returned error can't find the container with id c53677117f41be77fcf8f55d7bdeb666c878064e6a2d5d71d991b647c75c1f67 Jul 23 11:37:06.964846 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.964752 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5b46f9c-jrgj5" event={"ID":"35020295-870c-4c88-b2ab-12970419a2dd","Type":"ContainerStarted","Data":"01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f"} Jul 23 11:37:06.964846 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.964799 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5b46f9c-jrgj5" event={"ID":"35020295-870c-4c88-b2ab-12970419a2dd","Type":"ContainerStarted","Data":"c53677117f41be77fcf8f55d7bdeb666c878064e6a2d5d71d991b647c75c1f67"} Jul 23 11:37:06.999142 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:06.999097 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-d5b46f9c-jrgj5" podStartSLOduration=0.999080448 podStartE2EDuration="999.080448ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:06.997886855 +0000 UTC m=+75.103213267" watchObservedRunningTime="2026-07-23 11:37:06.999080448 +0000 UTC m=+75.104406860" Jul 23 11:37:07.972269 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:07.972176 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb"} Jul 23 11:37:07.972269 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:07.972260 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230"} Jul 23 11:37:07.972786 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:07.972299 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e"} Jul 23 11:37:07.972786 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:07.972338 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16"} Jul 23 11:37:07.972786 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:07.972376 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c"} Jul 23 11:37:08.979087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:08.979049 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerStarted","Data":"2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489"} Jul 23 11:37:09.009775 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:09.009721 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.64228109 podStartE2EDuration="7.009699977s" totalImagesPullingTime="4.343468515s" totalInitContainerRuntime="23.950372ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:37:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:37:09.008083147 +0000 UTC m=+77.113409557" watchObservedRunningTime="2026-07-23 11:37:09.009699977 +0000 UTC m=+77.115026388" Jul 23 11:37:16.376525 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:16.376380 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:16.376525 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:16.376469 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:16.381502 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:16.381475 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:17.011807 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:17.011779 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:37:17.066659 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:17.066605 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb7fcbcf6-kf82q"] Jul 23 11:37:32.992684 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:32.992602 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7d7c5d85bb-kvm6v" podUID="a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" containerName="console" containerID="cri-o://8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2" gracePeriod=15 Jul 23 11:37:33.230070 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.230046 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:37:33.320079 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320030 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-config\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320079 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320086 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-oauth-serving-cert\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320333 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320117 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-oauth-config\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320333 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320284 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-service-ca\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320447 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320337 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llfpj\" (UniqueName: \"kubernetes.io/projected/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-kube-api-access-llfpj\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320447 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320373 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-trusted-ca-bundle\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320555 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320450 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-serving-cert\") pod \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\" (UID: \"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3\") " Jul 23 11:37:33.320555 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320541 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-config" (OuterVolumeSpecName: "console-config") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:33.320688 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320614 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:33.320740 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320712 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-service-ca" (OuterVolumeSpecName: "service-ca") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:33.320819 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320803 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:33.320883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320824 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-oauth-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:33.320883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320837 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-service-ca\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:33.320883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.320842 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:33.322772 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.322746 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:33.322871 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.322785 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:33.322871 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.322808 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-kube-api-access-llfpj" (OuterVolumeSpecName: "kube-api-access-llfpj") pod "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" (UID: "a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3"). InnerVolumeSpecName "kube-api-access-llfpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:37:33.422106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.422062 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:33.422106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.422099 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-console-oauth-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:33.422106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.422111 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llfpj\" (UniqueName: \"kubernetes.io/projected/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-kube-api-access-llfpj\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:33.422106 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:33.422120 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3-trusted-ca-bundle\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:34.070862 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.070830 2554 generic.go:356] "Generic (PLEG): container finished" podID="a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" containerID="8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2" exitCode=0 Jul 23 11:37:34.071297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.070875 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7d7c5d85bb-kvm6v" Jul 23 11:37:34.071297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.070911 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c5d85bb-kvm6v" event={"ID":"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3","Type":"ContainerDied","Data":"8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2"} Jul 23 11:37:34.071297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.070965 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7d7c5d85bb-kvm6v" event={"ID":"a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3","Type":"ContainerDied","Data":"4fee0a794305f04cd93152bd3a978c83ce19cbcf36925fb2d41a5e295f67c586"} Jul 23 11:37:34.072264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.072250 2554 scope.go:117] "RemoveContainer" containerID="8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2" Jul 23 11:37:34.081030 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.081010 2554 scope.go:117] "RemoveContainer" containerID="8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2" Jul 23 11:37:34.081320 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:37:34.081301 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2\": container with ID starting with 8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2 not found: ID does not exist" containerID="8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2" Jul 23 11:37:34.081376 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.081331 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2"} err="failed to get container status \"8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2\": rpc error: code = NotFound desc = could not find container \"8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2\": container with ID starting with 8b2647f19be8e9adba26ee6545d45dd67912a13e4f9b505a3547dd6ec1a271f2 not found: ID does not exist" Jul 23 11:37:34.091383 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.091353 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7d7c5d85bb-kvm6v"] Jul 23 11:37:34.098196 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.098165 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7d7c5d85bb-kvm6v"] Jul 23 11:37:34.455876 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.453201 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" path="/var/lib/kubelet/pods/a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3/volumes" Jul 23 11:37:34.946305 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:34.946267 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-svsd4" Jul 23 11:37:42.088043 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.087977 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-fb7fcbcf6-kf82q" podUID="eb7d165e-e557-486c-9f64-e3f40fa0049d" containerName="console" containerID="cri-o://df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6" gracePeriod=15 Jul 23 11:37:42.327964 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.327933 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:37:42.385201 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385101 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txdv2\" (UniqueName: \"kubernetes.io/projected/eb7d165e-e557-486c-9f64-e3f40fa0049d-kube-api-access-txdv2\") pod \"eb7d165e-e557-486c-9f64-e3f40fa0049d\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " Jul 23 11:37:42.385201 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385148 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-config\") pod \"eb7d165e-e557-486c-9f64-e3f40fa0049d\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " Jul 23 11:37:42.385201 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385189 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-serving-cert\") pod \"eb7d165e-e557-486c-9f64-e3f40fa0049d\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " Jul 23 11:37:42.385466 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385210 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-service-ca\") pod \"eb7d165e-e557-486c-9f64-e3f40fa0049d\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " Jul 23 11:37:42.385466 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385244 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-oauth-serving-cert\") pod \"eb7d165e-e557-486c-9f64-e3f40fa0049d\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " Jul 23 11:37:42.385466 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385263 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-oauth-config\") pod \"eb7d165e-e557-486c-9f64-e3f40fa0049d\" (UID: \"eb7d165e-e557-486c-9f64-e3f40fa0049d\") " Jul 23 11:37:42.385748 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385721 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-config" (OuterVolumeSpecName: "console-config") pod "eb7d165e-e557-486c-9f64-e3f40fa0049d" (UID: "eb7d165e-e557-486c-9f64-e3f40fa0049d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:42.385748 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385729 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "eb7d165e-e557-486c-9f64-e3f40fa0049d" (UID: "eb7d165e-e557-486c-9f64-e3f40fa0049d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:42.385850 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.385738 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-service-ca" (OuterVolumeSpecName: "service-ca") pod "eb7d165e-e557-486c-9f64-e3f40fa0049d" (UID: "eb7d165e-e557-486c-9f64-e3f40fa0049d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:37:42.387784 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.387756 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "eb7d165e-e557-486c-9f64-e3f40fa0049d" (UID: "eb7d165e-e557-486c-9f64-e3f40fa0049d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:42.387898 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.387791 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb7d165e-e557-486c-9f64-e3f40fa0049d-kube-api-access-txdv2" (OuterVolumeSpecName: "kube-api-access-txdv2") pod "eb7d165e-e557-486c-9f64-e3f40fa0049d" (UID: "eb7d165e-e557-486c-9f64-e3f40fa0049d"). InnerVolumeSpecName "kube-api-access-txdv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:37:42.387898 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.387810 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "eb7d165e-e557-486c-9f64-e3f40fa0049d" (UID: "eb7d165e-e557-486c-9f64-e3f40fa0049d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:37:42.486029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.485991 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:42.486029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.486024 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:42.486029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.486037 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-service-ca\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:42.486263 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.486049 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/eb7d165e-e557-486c-9f64-e3f40fa0049d-oauth-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:42.486263 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.486060 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/eb7d165e-e557-486c-9f64-e3f40fa0049d-console-oauth-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:42.486263 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:42.486072 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txdv2\" (UniqueName: \"kubernetes.io/projected/eb7d165e-e557-486c-9f64-e3f40fa0049d-kube-api-access-txdv2\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:37:43.106302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.106266 2554 generic.go:356] "Generic (PLEG): container finished" podID="eb7d165e-e557-486c-9f64-e3f40fa0049d" containerID="df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6" exitCode=0 Jul 23 11:37:43.106747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.106310 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb7fcbcf6-kf82q" Jul 23 11:37:43.106747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.106360 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb7fcbcf6-kf82q" event={"ID":"eb7d165e-e557-486c-9f64-e3f40fa0049d","Type":"ContainerDied","Data":"df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6"} Jul 23 11:37:43.106747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.106414 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb7fcbcf6-kf82q" event={"ID":"eb7d165e-e557-486c-9f64-e3f40fa0049d","Type":"ContainerDied","Data":"8d8baec3476c0bfd3ed05fedeb6fb291e226c169a502c65b393f2cd2cc3ce641"} Jul 23 11:37:43.106747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.106435 2554 scope.go:117] "RemoveContainer" containerID="df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6" Jul 23 11:37:43.115540 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.115521 2554 scope.go:117] "RemoveContainer" containerID="df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6" Jul 23 11:37:43.115861 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:37:43.115840 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6\": container with ID starting with df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6 not found: ID does not exist" containerID="df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6" Jul 23 11:37:43.115946 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.115873 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6"} err="failed to get container status \"df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6\": rpc error: code = NotFound desc = could not find container \"df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6\": container with ID starting with df8687b20d3891b200df3c1580b676853a15fe3638027cad7136e0bf9dbe94d6 not found: ID does not exist" Jul 23 11:37:43.123138 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.123105 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb7fcbcf6-kf82q"] Jul 23 11:37:43.131712 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:43.131681 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fb7fcbcf6-kf82q"] Jul 23 11:37:44.451992 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:37:44.451951 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb7d165e-e557-486c-9f64-e3f40fa0049d" path="/var/lib/kubelet/pods/eb7d165e-e557-486c-9f64-e3f40fa0049d/volumes" Jul 23 11:38:21.660853 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.660818 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:38:21.661374 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.661155 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="alertmanager" containerID="cri-o://2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c" gracePeriod=120 Jul 23 11:38:21.661374 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.661237 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-metric" containerID="cri-o://228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb" gracePeriod=120 Jul 23 11:38:21.661374 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.661303 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="prom-label-proxy" containerID="cri-o://2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489" gracePeriod=120 Jul 23 11:38:21.661374 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.661243 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-web" containerID="cri-o://0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e" gracePeriod=120 Jul 23 11:38:21.661593 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.661304 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="config-reloader" containerID="cri-o://be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16" gracePeriod=120 Jul 23 11:38:21.661593 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:21.661290 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy" containerID="cri-o://229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230" gracePeriod=120 Jul 23 11:38:22.251336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251302 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489" exitCode=0 Jul 23 11:38:22.251336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251329 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230" exitCode=0 Jul 23 11:38:22.251336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251337 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e" exitCode=0 Jul 23 11:38:22.251336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251344 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16" exitCode=0 Jul 23 11:38:22.251601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251351 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c" exitCode=0 Jul 23 11:38:22.251601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251372 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489"} Jul 23 11:38:22.251601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251443 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230"} Jul 23 11:38:22.251601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251469 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e"} Jul 23 11:38:22.251601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251495 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16"} Jul 23 11:38:22.251601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.251520 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c"} Jul 23 11:38:22.907182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.907157 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:22.999306 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999211 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-web-config\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999306 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999281 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999317 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-main-db\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999345 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-main-tls\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999390 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-metrics-client-ca\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999428 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hghdh\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-kube-api-access-hghdh\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999453 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-cluster-tls-config\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999487 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999520 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-volume\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999554 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-tls-assets\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999648 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-out\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999680 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-web\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:22.999894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999712 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-metric\") pod \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\" (UID: \"5dd808d4-1bf0-4402-a28e-ed17d13613d4\") " Jul 23 11:38:23.000667 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999670 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:38:23.000667 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:22.999766 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:38:23.000667 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.000035 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:38:23.003868 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.003822 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.004442 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.004401 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:38:23.004736 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.004703 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.004862 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.004826 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-kube-api-access-hghdh" (OuterVolumeSpecName: "kube-api-access-hghdh") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "kube-api-access-hghdh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:38:23.004955 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.004938 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-volume" (OuterVolumeSpecName: "config-volume") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.005094 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.005065 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.005191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.005164 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.005393 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.005358 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-out" (OuterVolumeSpecName: "config-out") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:38:23.008552 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.008523 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.014848 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.014816 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-web-config" (OuterVolumeSpecName: "web-config") pod "5dd808d4-1bf0-4402-a28e-ed17d13613d4" (UID: "5dd808d4-1bf0-4402-a28e-ed17d13613d4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:38:23.101054 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101007 2554 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-metrics-client-ca\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101054 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101040 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hghdh\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-kube-api-access-hghdh\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101054 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101051 2554 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-cluster-tls-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101054 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101061 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101071 2554 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-volume\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101079 2554 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5dd808d4-1bf0-4402-a28e-ed17d13613d4-tls-assets\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101087 2554 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-config-out\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101095 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101103 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101112 2554 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-web-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101120 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101129 2554 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5dd808d4-1bf0-4402-a28e-ed17d13613d4-alertmanager-main-db\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.101307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.101139 2554 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5dd808d4-1bf0-4402-a28e-ed17d13613d4-secret-alertmanager-main-tls\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:38:23.259529 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.259447 2554 generic.go:356] "Generic (PLEG): container finished" podID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerID="228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb" exitCode=0 Jul 23 11:38:23.259529 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.259524 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb"} Jul 23 11:38:23.259734 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.259577 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.259734 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.259593 2554 scope.go:117] "RemoveContainer" containerID="2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489" Jul 23 11:38:23.259734 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.259583 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5dd808d4-1bf0-4402-a28e-ed17d13613d4","Type":"ContainerDied","Data":"741e0cad216aff516fee24b7a98358ce6c20dfa5af0be7f6d9b334840631d398"} Jul 23 11:38:23.267686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.267664 2554 scope.go:117] "RemoveContainer" containerID="228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb" Jul 23 11:38:23.274670 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.274648 2554 scope.go:117] "RemoveContainer" containerID="229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230" Jul 23 11:38:23.281700 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.281680 2554 scope.go:117] "RemoveContainer" containerID="0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e" Jul 23 11:38:23.285098 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.284849 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:38:23.290019 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.289992 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:38:23.292894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.292852 2554 scope.go:117] "RemoveContainer" containerID="be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16" Jul 23 11:38:23.300784 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.300762 2554 scope.go:117] "RemoveContainer" containerID="2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c" Jul 23 11:38:23.308156 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.308129 2554 scope.go:117] "RemoveContainer" containerID="a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9" Jul 23 11:38:23.315671 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.315648 2554 scope.go:117] "RemoveContainer" containerID="2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489" Jul 23 11:38:23.315956 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.315935 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489\": container with ID starting with 2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489 not found: ID does not exist" containerID="2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489" Jul 23 11:38:23.316006 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.315968 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489"} err="failed to get container status \"2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489\": rpc error: code = NotFound desc = could not find container \"2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489\": container with ID starting with 2d7a48e13f973cfe45c98ebfb53b563efb97243fd7f5f6161714a9797a69d489 not found: ID does not exist" Jul 23 11:38:23.316006 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.315994 2554 scope.go:117] "RemoveContainer" containerID="228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb" Jul 23 11:38:23.316247 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.316230 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb\": container with ID starting with 228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb not found: ID does not exist" containerID="228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb" Jul 23 11:38:23.316286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.316253 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb"} err="failed to get container status \"228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb\": rpc error: code = NotFound desc = could not find container \"228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb\": container with ID starting with 228777fd9443292d72ffb15973200a89a5cb6a9830b2e96cab05c72db9734deb not found: ID does not exist" Jul 23 11:38:23.316286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.316269 2554 scope.go:117] "RemoveContainer" containerID="229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230" Jul 23 11:38:23.316517 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.316501 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230\": container with ID starting with 229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230 not found: ID does not exist" containerID="229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230" Jul 23 11:38:23.316557 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.316520 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230"} err="failed to get container status \"229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230\": rpc error: code = NotFound desc = could not find container \"229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230\": container with ID starting with 229644af2e00184870ffc68eb5697f78195798dba86d5ac7ef347fb2aa66f230 not found: ID does not exist" Jul 23 11:38:23.316557 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.316533 2554 scope.go:117] "RemoveContainer" containerID="0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e" Jul 23 11:38:23.316801 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.316784 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e\": container with ID starting with 0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e not found: ID does not exist" containerID="0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e" Jul 23 11:38:23.316859 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.316805 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e"} err="failed to get container status \"0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e\": rpc error: code = NotFound desc = could not find container \"0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e\": container with ID starting with 0685691c9f99a57ee8084027e2c3e8ad12e37fc82ee47e4d135c5f7cc8a5146e not found: ID does not exist" Jul 23 11:38:23.316859 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.316837 2554 scope.go:117] "RemoveContainer" containerID="be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16" Jul 23 11:38:23.317037 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.317022 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16\": container with ID starting with be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16 not found: ID does not exist" containerID="be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16" Jul 23 11:38:23.317081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.317041 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16"} err="failed to get container status \"be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16\": rpc error: code = NotFound desc = could not find container \"be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16\": container with ID starting with be4ee90eb1e9a4197877733cb2c15b8ce856e543274971e650d13466849fab16 not found: ID does not exist" Jul 23 11:38:23.317081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.317054 2554 scope.go:117] "RemoveContainer" containerID="2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c" Jul 23 11:38:23.317231 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.317215 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c\": container with ID starting with 2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c not found: ID does not exist" containerID="2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c" Jul 23 11:38:23.317270 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.317234 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c"} err="failed to get container status \"2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c\": rpc error: code = NotFound desc = could not find container \"2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c\": container with ID starting with 2caf4743eb945ec01a7af71d37ae3b159f9a9763a69d475c435965ac47540e0c not found: ID does not exist" Jul 23 11:38:23.317270 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.317246 2554 scope.go:117] "RemoveContainer" containerID="a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9" Jul 23 11:38:23.317454 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:38:23.317437 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9\": container with ID starting with a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9 not found: ID does not exist" containerID="a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9" Jul 23 11:38:23.317499 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.317457 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9"} err="failed to get container status \"a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9\": rpc error: code = NotFound desc = could not find container \"a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9\": container with ID starting with a86cf32b97d6baf36a14de08bcd73d98dd8a92f25c25115b11842c589ac117e9 not found: ID does not exist" Jul 23 11:38:23.320862 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.320839 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:38:23.321116 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321104 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="init-config-reloader" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321118 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="init-config-reloader" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321125 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321130 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321137 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-metric" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321143 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-metric" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321151 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="config-reloader" Jul 23 11:38:23.321157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321156 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="config-reloader" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321163 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="alertmanager" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321169 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="alertmanager" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321178 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-web" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321183 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-web" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321189 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="prom-label-proxy" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321193 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="prom-label-proxy" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321200 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" containerName="console" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321204 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" containerName="console" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321211 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb7d165e-e557-486c-9f64-e3f40fa0049d" containerName="console" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321228 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb7d165e-e557-486c-9f64-e3f40fa0049d" containerName="console" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321266 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-web" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321274 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb7d165e-e557-486c-9f64-e3f40fa0049d" containerName="console" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321280 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a95c22fa-bb3c-4a43-9a3e-21ce7c2ab5e3" containerName="console" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321287 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321292 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="config-reloader" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321299 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="alertmanager" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321303 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="prom-label-proxy" Jul 23 11:38:23.321349 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.321309 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" containerName="kube-rbac-proxy-metric" Jul 23 11:38:23.329088 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.329052 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.332023 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.331994 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 23 11:38:23.332023 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.331995 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 23 11:38:23.332247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.331995 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 23 11:38:23.332247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.331995 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 23 11:38:23.332247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.331995 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 23 11:38:23.332360 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.332348 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wl6r9\"" Jul 23 11:38:23.332764 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.332749 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 23 11:38:23.332845 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.332770 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 23 11:38:23.333066 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.333051 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 23 11:38:23.337814 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.337792 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 23 11:38:23.343511 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.343482 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:38:23.503855 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.503817 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-web-config\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.503855 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.503851 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.503874 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.503891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7fhg\" (UniqueName: \"kubernetes.io/projected/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-kube-api-access-v7fhg\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.503980 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504018 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504046 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504068 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-config-out\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504091 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504108 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504168 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504197 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.504307 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.504247 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605174 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605139 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605174 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605181 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605404 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605303 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605404 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605357 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-web-config\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605517 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605401 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605517 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605448 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605517 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605476 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7fhg\" (UniqueName: \"kubernetes.io/projected/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-kube-api-access-v7fhg\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605517 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605507 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605726 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605565 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605726 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605605 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605726 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605610 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605726 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605700 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-config-out\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605902 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605738 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.605902 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.605762 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.606461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.606391 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.607367 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.607336 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.608542 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.608514 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-web-config\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.608662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.608575 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-config-volume\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.608662 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.608580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-config-out\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.608786 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.608696 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.608786 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.608702 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.608892 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.608784 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.609282 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.609263 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.609563 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.609547 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.610530 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.610509 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.615955 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.615936 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7fhg\" (UniqueName: \"kubernetes.io/projected/af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1-kube-api-access-v7fhg\") pod \"alertmanager-main-0\" (UID: \"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1\") " pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.639943 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.639912 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 23 11:38:23.784263 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:23.784182 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 23 11:38:23.786735 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:38:23.786702 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf2445a8_94ab_4b9e_8a4f_bbba7ebdebb1.slice/crio-ad971e87ec4bb6ce6c354864ed0d62fa4827d61d8a156dcbb9549e96aaeb1c2a WatchSource:0}: Error finding container ad971e87ec4bb6ce6c354864ed0d62fa4827d61d8a156dcbb9549e96aaeb1c2a: Status 404 returned error can't find the container with id ad971e87ec4bb6ce6c354864ed0d62fa4827d61d8a156dcbb9549e96aaeb1c2a Jul 23 11:38:24.265195 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:24.265092 2554 generic.go:356] "Generic (PLEG): container finished" podID="af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1" containerID="ff8a6b2dc2b4d236ddec5745671b0c47906f4c022f261cce891b83001e1ab344" exitCode=0 Jul 23 11:38:24.265644 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:24.265192 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerDied","Data":"ff8a6b2dc2b4d236ddec5745671b0c47906f4c022f261cce891b83001e1ab344"} Jul 23 11:38:24.265644 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:24.265280 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"ad971e87ec4bb6ce6c354864ed0d62fa4827d61d8a156dcbb9549e96aaeb1c2a"} Jul 23 11:38:24.452370 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:24.452339 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dd808d4-1bf0-4402-a28e-ed17d13613d4" path="/var/lib/kubelet/pods/5dd808d4-1bf0-4402-a28e-ed17d13613d4/volumes" Jul 23 11:38:25.273655 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.273601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"ff467944cdb07b8e9015c036ca1bf8472e5b699876b7b5bbbad4284b33175058"} Jul 23 11:38:25.274043 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.273719 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"ebbcfc81c17ae91515b099499c43fa69a8e978308a049b82530ae9a77b819345"} Jul 23 11:38:25.274043 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.273747 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"1a5ff4a8ec56717db7e5c9b791d412995b7817105df526137acb5893dd711320"} Jul 23 11:38:25.274043 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.273771 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"aad5d853a6cb78f80fa9a7e8268942735e171efe61bdcf196bb70764a3bc2bf4"} Jul 23 11:38:25.274043 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.273795 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"65f0c24dd48c157767332caff8340b31dc07586ddbba8aef039bc5b4aa999a0e"} Jul 23 11:38:25.274043 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.273817 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1","Type":"ContainerStarted","Data":"f21e37be3634b7cb6483c87382a37c6d6c96d2f9b3d06f01913879c85b54aa30"} Jul 23 11:38:25.303557 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.303504 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.287957485 podStartE2EDuration="2.303487965s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.53048ms" isStatefulPod=false podCreationTimestamp="2026-07-23 11:38:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:38:25.302588024 +0000 UTC m=+153.407914435" watchObservedRunningTime="2026-07-23 11:38:25.303487965 +0000 UTC m=+153.408814377" Jul 23 11:38:25.711092 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.711037 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69b9595b5d-hbd25"] Jul 23 11:38:25.713814 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.713790 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.717073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.716734 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 23 11:38:25.717073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.716953 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wtwhj\"" Jul 23 11:38:25.717324 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.717244 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 23 11:38:25.717324 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.717304 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 23 11:38:25.717432 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.717352 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 23 11:38:25.717486 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.717454 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 23 11:38:25.721527 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.721502 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 23 11:38:25.739242 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.739214 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69b9595b5d-hbd25"] Jul 23 11:38:25.825834 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.825799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-serving-certs-ca-bundle\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.825834 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.825841 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-telemeter-client-tls\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.826073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.825875 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-metrics-client-ca\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.826073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.825900 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-federate-client-tls\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.826073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.825925 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-secret-telemeter-client\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.826073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.825954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.826073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.826009 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.826286 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.826095 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhg5k\" (UniqueName: \"kubernetes.io/projected/0d80a1ac-163a-439b-9d8f-988144eabaf4-kube-api-access-zhg5k\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-metrics-client-ca\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-federate-client-tls\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927591 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927432 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-secret-telemeter-client\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927649 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927580 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927713 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927646 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927713 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927689 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhg5k\" (UniqueName: \"kubernetes.io/projected/0d80a1ac-163a-439b-9d8f-988144eabaf4-kube-api-access-zhg5k\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927808 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927770 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-serving-certs-ca-bundle\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.927808 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.927796 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-telemeter-client-tls\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.928470 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.928441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.928575 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.928486 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-metrics-client-ca\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.928803 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.928777 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0d80a1ac-163a-439b-9d8f-988144eabaf4-serving-certs-ca-bundle\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.930353 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.930331 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-telemeter-client-tls\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.930457 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.930352 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-secret-telemeter-client\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.930673 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.930654 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.931013 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.930992 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/0d80a1ac-163a-439b-9d8f-988144eabaf4-federate-client-tls\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:25.936267 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:25.936240 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhg5k\" (UniqueName: \"kubernetes.io/projected/0d80a1ac-163a-439b-9d8f-988144eabaf4-kube-api-access-zhg5k\") pod \"telemeter-client-69b9595b5d-hbd25\" (UID: \"0d80a1ac-163a-439b-9d8f-988144eabaf4\") " pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:26.025410 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:26.025317 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" Jul 23 11:38:26.182354 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:26.182257 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69b9595b5d-hbd25"] Jul 23 11:38:26.186157 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:38:26.186118 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d80a1ac_163a_439b_9d8f_988144eabaf4.slice/crio-be551f08c8f7e344e403c95d97aa815197eaa07d04654acd2c42b2ea11652a52 WatchSource:0}: Error finding container be551f08c8f7e344e403c95d97aa815197eaa07d04654acd2c42b2ea11652a52: Status 404 returned error can't find the container with id be551f08c8f7e344e403c95d97aa815197eaa07d04654acd2c42b2ea11652a52 Jul 23 11:38:26.280638 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:26.280513 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" event={"ID":"0d80a1ac-163a-439b-9d8f-988144eabaf4","Type":"ContainerStarted","Data":"be551f08c8f7e344e403c95d97aa815197eaa07d04654acd2c42b2ea11652a52"} Jul 23 11:38:29.296154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:29.296120 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" event={"ID":"0d80a1ac-163a-439b-9d8f-988144eabaf4","Type":"ContainerStarted","Data":"447f53e6e9acad6b724e9f3deefcda8afdd7ccd1293ce5a2965cacc7b1073c7d"} Jul 23 11:38:29.296566 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:29.296178 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" event={"ID":"0d80a1ac-163a-439b-9d8f-988144eabaf4","Type":"ContainerStarted","Data":"f03ea799da80765167816d10bdedf9ee929f7152bf56824fc499d13dd83aa4e0"} Jul 23 11:38:29.296566 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:29.296197 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" event={"ID":"0d80a1ac-163a-439b-9d8f-988144eabaf4","Type":"ContainerStarted","Data":"cf39703d8b8dfaa8ebc06081973bf2f5e455e7dd8b66b95479a49688944ce0d9"} Jul 23 11:38:29.321789 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:29.321732 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69b9595b5d-hbd25" podStartSLOduration=2.253942221 podStartE2EDuration="4.321715587s" totalImagesPullingTime="2.067773366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:38:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:38:29.321101109 +0000 UTC m=+157.426427514" watchObservedRunningTime="2026-07-23 11:38:29.321715587 +0000 UTC m=+157.427041999" Jul 23 11:38:30.131744 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.131707 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c584c48c5-fmrr7"] Jul 23 11:38:30.134888 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.134328 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.158161 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.158132 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c584c48c5-fmrr7"] Jul 23 11:38:30.265980 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.265935 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-serving-cert\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.265980 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.265980 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9s7d\" (UniqueName: \"kubernetes.io/projected/2069c5b4-9967-4d8d-ac73-751beff3e39b-kube-api-access-n9s7d\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.266234 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.266009 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-oauth-config\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.266234 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.266092 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-config\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.266234 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.266115 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-oauth-serving-cert\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.266234 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.266144 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-service-ca\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.266234 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.266164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-trusted-ca-bundle\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367383 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-config\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367423 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-oauth-serving-cert\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367957 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367445 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-service-ca\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367957 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367460 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-trusted-ca-bundle\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367957 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367508 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-serving-cert\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367957 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367525 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9s7d\" (UniqueName: \"kubernetes.io/projected/2069c5b4-9967-4d8d-ac73-751beff3e39b-kube-api-access-n9s7d\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.367957 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.367555 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-oauth-config\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.368208 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.368184 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-oauth-serving-cert\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.368362 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.368335 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-service-ca\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.368422 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.368354 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-config\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.368459 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.368425 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-trusted-ca-bundle\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.370590 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.370565 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-oauth-config\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.370807 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.370787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-serving-cert\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.380340 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.380316 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9s7d\" (UniqueName: \"kubernetes.io/projected/2069c5b4-9967-4d8d-ac73-751beff3e39b-kube-api-access-n9s7d\") pod \"console-7c584c48c5-fmrr7\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.445669 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.445550 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:30.582154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:30.582122 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c584c48c5-fmrr7"] Jul 23 11:38:30.585039 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:38:30.585011 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2069c5b4_9967_4d8d_ac73_751beff3e39b.slice/crio-3fc5d82c068ac5b4661ffa619b6ba07cf2d3697ac80de7cfa5f5901f217a218f WatchSource:0}: Error finding container 3fc5d82c068ac5b4661ffa619b6ba07cf2d3697ac80de7cfa5f5901f217a218f: Status 404 returned error can't find the container with id 3fc5d82c068ac5b4661ffa619b6ba07cf2d3697ac80de7cfa5f5901f217a218f Jul 23 11:38:31.305506 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:31.305471 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584c48c5-fmrr7" event={"ID":"2069c5b4-9967-4d8d-ac73-751beff3e39b","Type":"ContainerStarted","Data":"d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb"} Jul 23 11:38:31.305731 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:31.305521 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584c48c5-fmrr7" event={"ID":"2069c5b4-9967-4d8d-ac73-751beff3e39b","Type":"ContainerStarted","Data":"3fc5d82c068ac5b4661ffa619b6ba07cf2d3697ac80de7cfa5f5901f217a218f"} Jul 23 11:38:31.325782 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:31.325729 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c584c48c5-fmrr7" podStartSLOduration=1.325712492 podStartE2EDuration="1.325712492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:38:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:38:31.324700175 +0000 UTC m=+159.430026607" watchObservedRunningTime="2026-07-23 11:38:31.325712492 +0000 UTC m=+159.431038903" Jul 23 11:38:40.446348 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:40.446308 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:40.446762 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:40.446385 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:40.452011 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:40.451986 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:41.347935 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:41.347908 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:38:41.387511 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:38:41.387468 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d5b46f9c-jrgj5"] Jul 23 11:39:06.414681 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.414619 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-d5b46f9c-jrgj5" podUID="35020295-870c-4c88-b2ab-12970419a2dd" containerName="console" containerID="cri-o://01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f" gracePeriod=15 Jul 23 11:39:06.659925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.659887 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:39:06.762154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762056 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-trusted-ca-bundle\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762107 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f288b\" (UniqueName: \"kubernetes.io/projected/35020295-870c-4c88-b2ab-12970419a2dd-kube-api-access-f288b\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762139 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-serving-cert\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762155 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-console-config\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762481 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762182 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-service-ca\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762481 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762203 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-oauth-serving-cert\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762481 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762230 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-oauth-config\") pod \"35020295-870c-4c88-b2ab-12970419a2dd\" (UID: \"35020295-870c-4c88-b2ab-12970419a2dd\") " Jul 23 11:39:06.762690 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762662 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:39:06.762747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762699 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-service-ca" (OuterVolumeSpecName: "service-ca") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:39:06.762747 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762732 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:39:06.762830 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.762786 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-console-config" (OuterVolumeSpecName: "console-config") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:39:06.764514 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.764490 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35020295-870c-4c88-b2ab-12970419a2dd-kube-api-access-f288b" (OuterVolumeSpecName: "kube-api-access-f288b") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "kube-api-access-f288b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:39:06.765133 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.765103 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:39:06.765133 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.765116 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "35020295-870c-4c88-b2ab-12970419a2dd" (UID: "35020295-870c-4c88-b2ab-12970419a2dd"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:39:06.863460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863408 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-trusted-ca-bundle\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:06.863460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863455 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f288b\" (UniqueName: \"kubernetes.io/projected/35020295-870c-4c88-b2ab-12970419a2dd-kube-api-access-f288b\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:06.863460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863467 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:06.863460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863476 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-console-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:06.863460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863485 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-service-ca\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:06.863798 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863493 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/35020295-870c-4c88-b2ab-12970419a2dd-oauth-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:06.863798 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:06.863502 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/35020295-870c-4c88-b2ab-12970419a2dd-console-oauth-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:39:07.437906 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.437873 2554 generic.go:356] "Generic (PLEG): container finished" podID="35020295-870c-4c88-b2ab-12970419a2dd" containerID="01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f" exitCode=0 Jul 23 11:39:07.438322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.437922 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d5b46f9c-jrgj5" Jul 23 11:39:07.438322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.437954 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5b46f9c-jrgj5" event={"ID":"35020295-870c-4c88-b2ab-12970419a2dd","Type":"ContainerDied","Data":"01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f"} Jul 23 11:39:07.438322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.438004 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d5b46f9c-jrgj5" event={"ID":"35020295-870c-4c88-b2ab-12970419a2dd","Type":"ContainerDied","Data":"c53677117f41be77fcf8f55d7bdeb666c878064e6a2d5d71d991b647c75c1f67"} Jul 23 11:39:07.438322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.438024 2554 scope.go:117] "RemoveContainer" containerID="01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f" Jul 23 11:39:07.447272 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.447254 2554 scope.go:117] "RemoveContainer" containerID="01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f" Jul 23 11:39:07.447569 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:39:07.447550 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f\": container with ID starting with 01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f not found: ID does not exist" containerID="01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f" Jul 23 11:39:07.447649 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.447576 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f"} err="failed to get container status \"01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f\": rpc error: code = NotFound desc = could not find container \"01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f\": container with ID starting with 01001a54dcefc2b35b024454dbeacc7691638db81ccff0502589ed5b11eaf02f not found: ID does not exist" Jul 23 11:39:07.460544 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.460506 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d5b46f9c-jrgj5"] Jul 23 11:39:07.464951 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:07.464925 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d5b46f9c-jrgj5"] Jul 23 11:39:08.451477 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:08.451438 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35020295-870c-4c88-b2ab-12970419a2dd" path="/var/lib/kubelet/pods/35020295-870c-4c88-b2ab-12970419a2dd/volumes" Jul 23 11:39:38.218866 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.218836 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7975bbc757-ldhlz"] Jul 23 11:39:38.219312 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.219107 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35020295-870c-4c88-b2ab-12970419a2dd" containerName="console" Jul 23 11:39:38.219312 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.219117 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="35020295-870c-4c88-b2ab-12970419a2dd" containerName="console" Jul 23 11:39:38.219312 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.219176 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="35020295-870c-4c88-b2ab-12970419a2dd" containerName="console" Jul 23 11:39:38.222232 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.222212 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.236828 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.236800 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7975bbc757-ldhlz"] Jul 23 11:39:38.308385 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308346 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-serving-cert\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.308573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308398 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-trusted-ca-bundle\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.308573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308430 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-oauth-serving-cert\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.308573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308476 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-service-ca\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.308573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308510 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-config\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.308573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308540 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5499h\" (UniqueName: \"kubernetes.io/projected/5e019307-72fd-4edf-bad3-1ea2e4c0e683-kube-api-access-5499h\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.308573 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.308563 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-oauth-config\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.408890 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.408864 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-serving-cert\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.408973 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.408902 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-trusted-ca-bundle\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.408973 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.408923 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-oauth-serving-cert\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.408973 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.408962 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-service-ca\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.409095 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.408991 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-config\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.409095 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.409033 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5499h\" (UniqueName: \"kubernetes.io/projected/5e019307-72fd-4edf-bad3-1ea2e4c0e683-kube-api-access-5499h\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.409095 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.409061 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-oauth-config\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.409818 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.409788 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-config\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.409927 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.409843 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-service-ca\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.409927 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.409861 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-trusted-ca-bundle\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.410090 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.410066 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-oauth-serving-cert\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.411556 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.411526 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-serving-cert\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.411695 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.411674 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-oauth-config\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.418193 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.418170 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5499h\" (UniqueName: \"kubernetes.io/projected/5e019307-72fd-4edf-bad3-1ea2e4c0e683-kube-api-access-5499h\") pod \"console-7975bbc757-ldhlz\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.530509 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.530424 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:38.662376 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:38.662352 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7975bbc757-ldhlz"] Jul 23 11:39:38.664809 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:39:38.664784 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e019307_72fd_4edf_bad3_1ea2e4c0e683.slice/crio-cd87e45043d9973848c45c28feb05c3fc76fc6ff7dde248ef05f3722be7efd68 WatchSource:0}: Error finding container cd87e45043d9973848c45c28feb05c3fc76fc6ff7dde248ef05f3722be7efd68: Status 404 returned error can't find the container with id cd87e45043d9973848c45c28feb05c3fc76fc6ff7dde248ef05f3722be7efd68 Jul 23 11:39:39.564124 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:39.564084 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975bbc757-ldhlz" event={"ID":"5e019307-72fd-4edf-bad3-1ea2e4c0e683","Type":"ContainerStarted","Data":"5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762"} Jul 23 11:39:39.564505 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:39.564145 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975bbc757-ldhlz" event={"ID":"5e019307-72fd-4edf-bad3-1ea2e4c0e683","Type":"ContainerStarted","Data":"cd87e45043d9973848c45c28feb05c3fc76fc6ff7dde248ef05f3722be7efd68"} Jul 23 11:39:39.588939 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:39.588886 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7975bbc757-ldhlz" podStartSLOduration=1.588867521 podStartE2EDuration="1.588867521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:39:39.587405025 +0000 UTC m=+227.692731449" watchObservedRunningTime="2026-07-23 11:39:39.588867521 +0000 UTC m=+227.694193932" Jul 23 11:39:48.530740 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:48.530702 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:48.531124 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:48.530768 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:48.535690 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:48.535671 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:48.603703 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:48.603679 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:39:48.652185 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:39:48.652155 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c584c48c5-fmrr7"] Jul 23 11:40:13.673940 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:13.673848 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7c584c48c5-fmrr7" podUID="2069c5b4-9967-4d8d-ac73-751beff3e39b" containerName="console" containerID="cri-o://d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb" gracePeriod=15 Jul 23 11:40:13.917400 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:13.917377 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:40:14.005791 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005697 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-trusted-ca-bundle\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.005791 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005750 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-serving-cert\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.005791 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005793 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n9s7d\" (UniqueName: \"kubernetes.io/projected/2069c5b4-9967-4d8d-ac73-751beff3e39b-kube-api-access-n9s7d\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.006051 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005824 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-oauth-config\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.006051 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005874 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-config\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.006051 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005903 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-service-ca\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.006051 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.005971 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-oauth-serving-cert\") pod \"2069c5b4-9967-4d8d-ac73-751beff3e39b\" (UID: \"2069c5b4-9967-4d8d-ac73-751beff3e39b\") " Jul 23 11:40:14.006316 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.006178 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:40:14.006416 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.006379 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-config" (OuterVolumeSpecName: "console-config") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:40:14.006416 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.006388 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-service-ca" (OuterVolumeSpecName: "service-ca") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:40:14.006582 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.006427 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:40:14.008217 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.008195 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2069c5b4-9967-4d8d-ac73-751beff3e39b-kube-api-access-n9s7d" (OuterVolumeSpecName: "kube-api-access-n9s7d") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "kube-api-access-n9s7d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:40:14.008293 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.008195 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:40:14.008293 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.008263 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2069c5b4-9967-4d8d-ac73-751beff3e39b" (UID: "2069c5b4-9967-4d8d-ac73-751beff3e39b"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:40:14.107567 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107525 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-trusted-ca-bundle\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.107567 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107557 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.107567 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107570 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9s7d\" (UniqueName: \"kubernetes.io/projected/2069c5b4-9967-4d8d-ac73-751beff3e39b-kube-api-access-n9s7d\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.107883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107582 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-oauth-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.107883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107593 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-console-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.107883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107604 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-service-ca\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.107883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.107615 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2069c5b4-9967-4d8d-ac73-751beff3e39b-oauth-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:40:14.692493 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.692462 2554 generic.go:356] "Generic (PLEG): container finished" podID="2069c5b4-9967-4d8d-ac73-751beff3e39b" containerID="d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb" exitCode=0 Jul 23 11:40:14.692493 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.692498 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584c48c5-fmrr7" event={"ID":"2069c5b4-9967-4d8d-ac73-751beff3e39b","Type":"ContainerDied","Data":"d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb"} Jul 23 11:40:14.693023 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.692519 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c584c48c5-fmrr7" Jul 23 11:40:14.693023 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.692540 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c584c48c5-fmrr7" event={"ID":"2069c5b4-9967-4d8d-ac73-751beff3e39b","Type":"ContainerDied","Data":"3fc5d82c068ac5b4661ffa619b6ba07cf2d3697ac80de7cfa5f5901f217a218f"} Jul 23 11:40:14.693023 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.692560 2554 scope.go:117] "RemoveContainer" containerID="d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb" Jul 23 11:40:14.701102 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.701086 2554 scope.go:117] "RemoveContainer" containerID="d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb" Jul 23 11:40:14.701348 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:40:14.701327 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb\": container with ID starting with d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb not found: ID does not exist" containerID="d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb" Jul 23 11:40:14.701401 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.701357 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb"} err="failed to get container status \"d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb\": rpc error: code = NotFound desc = could not find container \"d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb\": container with ID starting with d7d8963b4f87869a514d032e7308f0e1949f1b6e9bc19730de34f0e85b1f4ffb not found: ID does not exist" Jul 23 11:40:14.709156 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.709131 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7c584c48c5-fmrr7"] Jul 23 11:40:14.716974 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:14.714903 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7c584c48c5-fmrr7"] Jul 23 11:40:15.033248 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.033176 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7"] Jul 23 11:40:15.033496 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.033484 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2069c5b4-9967-4d8d-ac73-751beff3e39b" containerName="console" Jul 23 11:40:15.033540 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.033498 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="2069c5b4-9967-4d8d-ac73-751beff3e39b" containerName="console" Jul 23 11:40:15.033581 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.033572 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="2069c5b4-9967-4d8d-ac73-751beff3e39b" containerName="console" Jul 23 11:40:15.037533 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.037513 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.040146 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.040127 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 23 11:40:15.040262 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.040142 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-z6cs2\"" Jul 23 11:40:15.040262 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.040148 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 23 11:40:15.040989 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.040969 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 23 11:40:15.041086 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.040990 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 23 11:40:15.047560 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.047541 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7"] Jul 23 11:40:15.119894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.119858 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qhzs\" (UniqueName: \"kubernetes.io/projected/2affc7c0-4142-48dc-ba7d-1928e9ed1125-kube-api-access-8qhzs\") pod \"managed-serviceaccount-addon-agent-76764dbd7b-6zlj7\" (UID: \"2affc7c0-4142-48dc-ba7d-1928e9ed1125\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.120061 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.119905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2affc7c0-4142-48dc-ba7d-1928e9ed1125-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76764dbd7b-6zlj7\" (UID: \"2affc7c0-4142-48dc-ba7d-1928e9ed1125\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.143535 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.143501 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq"] Jul 23 11:40:15.146767 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.146746 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.149675 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.149641 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 23 11:40:15.149812 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.149718 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 23 11:40:15.149812 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.149736 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 23 11:40:15.149812 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.149719 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 23 11:40:15.160373 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.160347 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq"] Jul 23 11:40:15.221285 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.221244 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qhzs\" (UniqueName: \"kubernetes.io/projected/2affc7c0-4142-48dc-ba7d-1928e9ed1125-kube-api-access-8qhzs\") pod \"managed-serviceaccount-addon-agent-76764dbd7b-6zlj7\" (UID: \"2affc7c0-4142-48dc-ba7d-1928e9ed1125\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.221462 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.221290 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2affc7c0-4142-48dc-ba7d-1928e9ed1125-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76764dbd7b-6zlj7\" (UID: \"2affc7c0-4142-48dc-ba7d-1928e9ed1125\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.223979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.223958 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/2affc7c0-4142-48dc-ba7d-1928e9ed1125-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-76764dbd7b-6zlj7\" (UID: \"2affc7c0-4142-48dc-ba7d-1928e9ed1125\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.231100 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.231079 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qhzs\" (UniqueName: \"kubernetes.io/projected/2affc7c0-4142-48dc-ba7d-1928e9ed1125-kube-api-access-8qhzs\") pod \"managed-serviceaccount-addon-agent-76764dbd7b-6zlj7\" (UID: \"2affc7c0-4142-48dc-ba7d-1928e9ed1125\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.322071 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.322034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.322248 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.322078 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.322248 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.322105 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2ljs\" (UniqueName: \"kubernetes.io/projected/dd7c37e9-2ac1-43cd-891a-07004f119876-kube-api-access-p2ljs\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.322248 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.322195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-ca\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.322248 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.322234 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-hub\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.322388 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.322313 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7c37e9-2ac1-43cd-891a-07004f119876-additional-service-ca\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.358137 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.358099 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" Jul 23 11:40:15.423351 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.423315 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7c37e9-2ac1-43cd-891a-07004f119876-additional-service-ca\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.423494 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.423361 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.423494 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.423382 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.423494 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.423416 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2ljs\" (UniqueName: \"kubernetes.io/projected/dd7c37e9-2ac1-43cd-891a-07004f119876-kube-api-access-p2ljs\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.423494 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.423457 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-ca\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.423645 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.423493 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-hub\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.424270 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.424164 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd7c37e9-2ac1-43cd-891a-07004f119876-additional-service-ca\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.426180 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.426150 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-hub\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.426485 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.426465 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.426544 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.426532 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-ca\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.426871 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.426853 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dd7c37e9-2ac1-43cd-891a-07004f119876-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.432461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.432432 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2ljs\" (UniqueName: \"kubernetes.io/projected/dd7c37e9-2ac1-43cd-891a-07004f119876-kube-api-access-p2ljs\") pod \"cluster-proxy-proxy-agent-544487fd8b-vgdkq\" (UID: \"dd7c37e9-2ac1-43cd-891a-07004f119876\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.455877 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.455846 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" Jul 23 11:40:15.490078 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.489970 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7"] Jul 23 11:40:15.591206 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.591123 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq"] Jul 23 11:40:15.593873 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:40:15.593849 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddd7c37e9_2ac1_43cd_891a_07004f119876.slice/crio-0db3190dadc73050e113b82ad4dd1a9eaf4f478f32710913d1523ac6d3512e67 WatchSource:0}: Error finding container 0db3190dadc73050e113b82ad4dd1a9eaf4f478f32710913d1523ac6d3512e67: Status 404 returned error can't find the container with id 0db3190dadc73050e113b82ad4dd1a9eaf4f478f32710913d1523ac6d3512e67 Jul 23 11:40:15.697931 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.697894 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" event={"ID":"dd7c37e9-2ac1-43cd-891a-07004f119876","Type":"ContainerStarted","Data":"0db3190dadc73050e113b82ad4dd1a9eaf4f478f32710913d1523ac6d3512e67"} Jul 23 11:40:15.698921 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:15.698888 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" event={"ID":"2affc7c0-4142-48dc-ba7d-1928e9ed1125","Type":"ContainerStarted","Data":"306cc5fd70f29543d53fa07e33b3f142abec8a9c291d803c021cf943346368ac"} Jul 23 11:40:16.451799 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:16.451763 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2069c5b4-9967-4d8d-ac73-751beff3e39b" path="/var/lib/kubelet/pods/2069c5b4-9967-4d8d-ac73-751beff3e39b/volumes" Jul 23 11:40:20.725389 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:20.725353 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" event={"ID":"dd7c37e9-2ac1-43cd-891a-07004f119876","Type":"ContainerStarted","Data":"cab0e393583b999c45c533fa753563413ad217fd148ab097f45110260b6a88fe"} Jul 23 11:40:20.725890 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:20.725401 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" event={"ID":"dd7c37e9-2ac1-43cd-891a-07004f119876","Type":"ContainerStarted","Data":"13c087707110d97568fb491536ac33ea94fd2a7d90f636cba4eb8382b5dd7bfc"} Jul 23 11:40:20.725890 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:20.725418 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" event={"ID":"dd7c37e9-2ac1-43cd-891a-07004f119876","Type":"ContainerStarted","Data":"ea1f707a94dd4a77c95fe1282bf031f4e3014a906ba8bf3f5941595d911fa516"} Jul 23 11:40:20.726760 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:20.726736 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" event={"ID":"2affc7c0-4142-48dc-ba7d-1928e9ed1125","Type":"ContainerStarted","Data":"fc2598559109ce6265584503fdcf1a3073bd09176c01a9aed37fa216ca9e75b2"} Jul 23 11:40:20.750753 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:20.750709 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-544487fd8b-vgdkq" podStartSLOduration=1.548482879 podStartE2EDuration="5.750693613s" totalImagesPullingTime="4.202210734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:40:20.749839247 +0000 UTC m=+268.855165662" watchObservedRunningTime="2026-07-23 11:40:20.750693613 +0000 UTC m=+268.856020025" Jul 23 11:40:20.767600 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:20.767561 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-76764dbd7b-6zlj7" podStartSLOduration=1.479826273 podStartE2EDuration="5.767548215s" totalImagesPullingTime="4.287721942s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:40:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:40:20.766129948 +0000 UTC m=+268.871456370" watchObservedRunningTime="2026-07-23 11:40:20.767548215 +0000 UTC m=+268.872874627" Jul 23 11:40:52.294461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:40:52.294432 2554 kubelet.go:1648] "Image garbage collection succeeded" Jul 23 11:41:06.684314 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.684279 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r"] Jul 23 11:41:06.686225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.686209 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.688994 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.688973 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 23 11:41:06.689580 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.689567 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 23 11:41:06.689640 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.689568 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-k9v9z\"" Jul 23 11:41:06.696546 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.696525 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r"] Jul 23 11:41:06.840849 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.840807 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06eeb81e-66ac-4662-a5f0-cf9c8fabb332-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gb65r\" (UID: \"06eeb81e-66ac-4662-a5f0-cf9c8fabb332\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.841036 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.840891 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jvjz\" (UniqueName: \"kubernetes.io/projected/06eeb81e-66ac-4662-a5f0-cf9c8fabb332-kube-api-access-2jvjz\") pod \"openshift-lws-operator-fd8ccff4c-gb65r\" (UID: \"06eeb81e-66ac-4662-a5f0-cf9c8fabb332\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.941467 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.941389 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jvjz\" (UniqueName: \"kubernetes.io/projected/06eeb81e-66ac-4662-a5f0-cf9c8fabb332-kube-api-access-2jvjz\") pod \"openshift-lws-operator-fd8ccff4c-gb65r\" (UID: \"06eeb81e-66ac-4662-a5f0-cf9c8fabb332\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.941467 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.941430 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06eeb81e-66ac-4662-a5f0-cf9c8fabb332-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gb65r\" (UID: \"06eeb81e-66ac-4662-a5f0-cf9c8fabb332\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.941779 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.941760 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/06eeb81e-66ac-4662-a5f0-cf9c8fabb332-tmp\") pod \"openshift-lws-operator-fd8ccff4c-gb65r\" (UID: \"06eeb81e-66ac-4662-a5f0-cf9c8fabb332\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.950861 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.950839 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jvjz\" (UniqueName: \"kubernetes.io/projected/06eeb81e-66ac-4662-a5f0-cf9c8fabb332-kube-api-access-2jvjz\") pod \"openshift-lws-operator-fd8ccff4c-gb65r\" (UID: \"06eeb81e-66ac-4662-a5f0-cf9c8fabb332\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:06.995448 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:06.995410 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" Jul 23 11:41:07.119670 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:07.119645 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r"] Jul 23 11:41:07.122674 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:41:07.122643 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06eeb81e_66ac_4662_a5f0_cf9c8fabb332.slice/crio-1d871c9dc8125df8a928e28aa9a7aeb88ce755d4022acf5fd90db09804568eae WatchSource:0}: Error finding container 1d871c9dc8125df8a928e28aa9a7aeb88ce755d4022acf5fd90db09804568eae: Status 404 returned error can't find the container with id 1d871c9dc8125df8a928e28aa9a7aeb88ce755d4022acf5fd90db09804568eae Jul 23 11:41:07.124606 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:07.124589 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:41:07.911107 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:07.911047 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" event={"ID":"06eeb81e-66ac-4662-a5f0-cf9c8fabb332","Type":"ContainerStarted","Data":"1d871c9dc8125df8a928e28aa9a7aeb88ce755d4022acf5fd90db09804568eae"} Jul 23 11:41:09.921784 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:09.921689 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" event={"ID":"06eeb81e-66ac-4662-a5f0-cf9c8fabb332","Type":"ContainerStarted","Data":"c74ac935897ea91bf75eb96b779e3f151304ab5300f7f5287b6cf21c37348384"} Jul 23 11:41:09.940279 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:09.940218 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-gb65r" podStartSLOduration=1.478208175 podStartE2EDuration="3.940196308s" totalImagesPullingTime="2.461988133s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:09.9385775 +0000 UTC m=+318.043903922" watchObservedRunningTime="2026-07-23 11:41:09.940196308 +0000 UTC m=+318.045522723" Jul 23 11:41:30.393154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.393118 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77"] Jul 23 11:41:30.396731 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.396714 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.408324 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:30.408286 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" type="*v1.Secret" Jul 23 11:41:30.408324 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:30.408286 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"opendatahub-operator-controller-manager-service-cert\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" Jul 23 11:41:30.408725 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.408707 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 23 11:41:30.408829 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.408748 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 23 11:41:30.408829 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.408760 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-rwjxf\"" Jul 23 11:41:30.442986 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.442955 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57c365ab-3761-4a8c-9296-8eda6f086567-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.443151 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.443002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.443151 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.443032 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.443151 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.443086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c365ab-3761-4a8c-9296-8eda6f086567-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.443151 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.443134 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brbqn\" (UniqueName: \"kubernetes.io/projected/57c365ab-3761-4a8c-9296-8eda6f086567-kube-api-access-brbqn\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.505131 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.505098 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77"] Jul 23 11:41:30.543903 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.543870 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.544104 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.543913 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.544104 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.544048 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c365ab-3761-4a8c-9296-8eda6f086567-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.544222 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.544106 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brbqn\" (UniqueName: \"kubernetes.io/projected/57c365ab-3761-4a8c-9296-8eda6f086567-kube-api-access-brbqn\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.544222 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.544194 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57c365ab-3761-4a8c-9296-8eda6f086567-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.544505 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.544488 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/57c365ab-3761-4a8c-9296-8eda6f086567-manifests\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.546443 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.546421 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57c365ab-3761-4a8c-9296-8eda6f086567-tmp\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:30.564287 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:30.564254 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brbqn\" (UniqueName: \"kubernetes.io/projected/57c365ab-3761-4a8c-9296-8eda6f086567-kube-api-access-brbqn\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:31.529135 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:31.529105 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 23 11:41:31.544613 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:31.544588 2554 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 23 11:41:31.544731 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:31.544595 2554 secret.go:189] Couldn't get secret opendatahub/opendatahub-operator-controller-manager-service-cert: failed to sync secret cache: timed out waiting for the condition Jul 23 11:41:31.544731 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:31.544685 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-apiservice-cert podName:57c365ab-3761-4a8c-9296-8eda6f086567 nodeName:}" failed. No retries permitted until 2026-07-23 11:41:32.044666551 +0000 UTC m=+340.149992940 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "apiservice-cert" (UniqueName: "kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-apiservice-cert") pod "opendatahub-operator-controller-manager-7c7646c6cc-n2g77" (UID: "57c365ab-3761-4a8c-9296-8eda6f086567") : failed to sync secret cache: timed out waiting for the condition Jul 23 11:41:31.544804 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:31.544735 2554 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-webhook-cert podName:57c365ab-3761-4a8c-9296-8eda6f086567 nodeName:}" failed. No retries permitted until 2026-07-23 11:41:32.044719615 +0000 UTC m=+340.150046004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-webhook-cert") pod "opendatahub-operator-controller-manager-7c7646c6cc-n2g77" (UID: "57c365ab-3761-4a8c-9296-8eda6f086567") : failed to sync secret cache: timed out waiting for the condition Jul 23 11:41:31.829087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:31.829054 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 23 11:41:32.055509 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:32.055474 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:32.055509 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:32.055513 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:32.058062 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:32.058032 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:32.058062 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:32.058043 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/57c365ab-3761-4a8c-9296-8eda6f086567-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c7646c6cc-n2g77\" (UID: \"57c365ab-3761-4a8c-9296-8eda6f086567\") " pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:32.207251 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:32.207149 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:32.365461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:32.365435 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77"] Jul 23 11:41:32.368387 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:41:32.368357 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c365ab_3761_4a8c_9296_8eda6f086567.slice/crio-8cb9403207802c60f709f11fb85f3b250c35d53b23feaf48dc70a617a62fc82c WatchSource:0}: Error finding container 8cb9403207802c60f709f11fb85f3b250c35d53b23feaf48dc70a617a62fc82c: Status 404 returned error can't find the container with id 8cb9403207802c60f709f11fb85f3b250c35d53b23feaf48dc70a617a62fc82c Jul 23 11:41:33.021050 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:33.020929 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" event={"ID":"57c365ab-3761-4a8c-9296-8eda6f086567","Type":"ContainerStarted","Data":"8cb9403207802c60f709f11fb85f3b250c35d53b23feaf48dc70a617a62fc82c"} Jul 23 11:41:36.036148 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:36.036111 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" event={"ID":"57c365ab-3761-4a8c-9296-8eda6f086567","Type":"ContainerStarted","Data":"d240391327aed09b1cb0a6cb2600c3866991dbd47e657c06f9443deb4941810c"} Jul 23 11:41:37.041990 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:37.041954 2554 generic.go:356] "Generic (PLEG): container finished" podID="57c365ab-3761-4a8c-9296-8eda6f086567" containerID="d240391327aed09b1cb0a6cb2600c3866991dbd47e657c06f9443deb4941810c" exitCode=0 Jul 23 11:41:37.042401 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:37.042050 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" event={"ID":"57c365ab-3761-4a8c-9296-8eda6f086567","Type":"ContainerDied","Data":"d240391327aed09b1cb0a6cb2600c3866991dbd47e657c06f9443deb4941810c"} Jul 23 11:41:38.048490 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:38.048451 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" event={"ID":"57c365ab-3761-4a8c-9296-8eda6f086567","Type":"ContainerStarted","Data":"83dd8061c489847496cd0411d6e42713ec6fdbe25e8d806ed0070f38865f3bec"} Jul 23 11:41:38.048984 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:38.048591 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:38.074773 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:38.074719 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" podStartSLOduration=3.768587621 podStartE2EDuration="8.074698271s" totalImagesPullingTime="3.293481708s" totalInitContainerRuntime="1.012628942s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:38.072112042 +0000 UTC m=+346.177438455" watchObservedRunningTime="2026-07-23 11:41:38.074698271 +0000 UTC m=+346.180024677" Jul 23 11:41:49.055413 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:49.055338 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c7646c6cc-n2g77" Jul 23 11:41:51.387918 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.387886 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr"] Jul 23 11:41:51.391315 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.391293 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.396230 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.396208 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 11:41:51.396506 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.396490 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 23 11:41:51.397275 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.397258 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 23 11:41:51.397468 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.397455 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 23 11:41:51.411927 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.411879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr"] Jul 23 11:41:51.526934 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.526894 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd86c0-746c-46e9-a026-487b5d3f81ff-tls-certs\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.527105 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.526952 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89pc\" (UniqueName: \"kubernetes.io/projected/3cfd86c0-746c-46e9-a026-487b5d3f81ff-kube-api-access-h89pc\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.527105 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.527012 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cfd86c0-746c-46e9-a026-487b5d3f81ff-tmp\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.587085 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.587047 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9cwng"] Jul 23 11:41:51.590390 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.590370 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.593100 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:51.593069 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-dockercfg-7jwk6\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7jwk6\"" type="*v1.Secret" Jul 23 11:41:51.593422 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:51.593404 2554 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"odh-model-controller-webhook-cert\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" type="*v1.Secret" Jul 23 11:41:51.593466 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:41:51.593420 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"odh-model-controller-7998c67dc4-9cwng\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="de982565-fe14-4b51-84fa-3ff6144a1617" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.619797 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.619758 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9cwng"] Jul 23 11:41:51.627600 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.627563 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89pc\" (UniqueName: \"kubernetes.io/projected/3cfd86c0-746c-46e9-a026-487b5d3f81ff-kube-api-access-h89pc\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.627805 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.627675 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cfd86c0-746c-46e9-a026-487b5d3f81ff-tmp\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.627805 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.627732 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd86c0-746c-46e9-a026-487b5d3f81ff-tls-certs\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.630732 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.630705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3cfd86c0-746c-46e9-a026-487b5d3f81ff-tls-certs\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.631261 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.631231 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3cfd86c0-746c-46e9-a026-487b5d3f81ff-tmp\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.663591 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.663521 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89pc\" (UniqueName: \"kubernetes.io/projected/3cfd86c0-746c-46e9-a026-487b5d3f81ff-kube-api-access-h89pc\") pod \"kube-auth-proxy-5b7f786dc8-zxjwr\" (UID: \"3cfd86c0-746c-46e9-a026-487b5d3f81ff\") " pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.701425 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.701398 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" Jul 23 11:41:51.728874 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.728843 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkb45\" (UniqueName: \"kubernetes.io/projected/de982565-fe14-4b51-84fa-3ff6144a1617-kube-api-access-pkb45\") pod \"odh-model-controller-7998c67dc4-9cwng\" (UID: \"de982565-fe14-4b51-84fa-3ff6144a1617\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.729002 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.728898 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de982565-fe14-4b51-84fa-3ff6144a1617-cert\") pod \"odh-model-controller-7998c67dc4-9cwng\" (UID: \"de982565-fe14-4b51-84fa-3ff6144a1617\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.829993 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.829952 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkb45\" (UniqueName: \"kubernetes.io/projected/de982565-fe14-4b51-84fa-3ff6144a1617-kube-api-access-pkb45\") pod \"odh-model-controller-7998c67dc4-9cwng\" (UID: \"de982565-fe14-4b51-84fa-3ff6144a1617\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.830196 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.830005 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de982565-fe14-4b51-84fa-3ff6144a1617-cert\") pod \"odh-model-controller-7998c67dc4-9cwng\" (UID: \"de982565-fe14-4b51-84fa-3ff6144a1617\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.850514 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.850477 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkb45\" (UniqueName: \"kubernetes.io/projected/de982565-fe14-4b51-84fa-3ff6144a1617-kube-api-access-pkb45\") pod \"odh-model-controller-7998c67dc4-9cwng\" (UID: \"de982565-fe14-4b51-84fa-3ff6144a1617\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:51.870241 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:51.870205 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr"] Jul 23 11:41:51.876228 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:41:51.876191 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cfd86c0_746c_46e9_a026_487b5d3f81ff.slice/crio-7c60d879e1a3727f12aa38eb6666a5e5ee9586a1a0d0265ce92bba9ea0a6e327 WatchSource:0}: Error finding container 7c60d879e1a3727f12aa38eb6666a5e5ee9586a1a0d0265ce92bba9ea0a6e327: Status 404 returned error can't find the container with id 7c60d879e1a3727f12aa38eb6666a5e5ee9586a1a0d0265ce92bba9ea0a6e327 Jul 23 11:41:52.106616 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.106576 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" event={"ID":"3cfd86c0-746c-46e9-a026-487b5d3f81ff","Type":"ContainerStarted","Data":"7c60d879e1a3727f12aa38eb6666a5e5ee9586a1a0d0265ce92bba9ea0a6e327"} Jul 23 11:41:52.684886 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.684850 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dlqj6"] Jul 23 11:41:52.689537 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.689515 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.692461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.692440 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 23 11:41:52.703870 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.703839 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dlqj6"] Jul 23 11:41:52.717357 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.717327 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 23 11:41:52.723194 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.723163 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/de982565-fe14-4b51-84fa-3ff6144a1617-cert\") pod \"odh-model-controller-7998c67dc4-9cwng\" (UID: \"de982565-fe14-4b51-84fa-3ff6144a1617\") " pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:52.738937 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.738903 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxldr\" (UniqueName: \"kubernetes.io/projected/28b25b2d-cd54-4ac4-8f63-531946cef94e-kube-api-access-rxldr\") pod \"model-serving-api-54d89bd5fc-dlqj6\" (UID: \"28b25b2d-cd54-4ac4-8f63-531946cef94e\") " pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.739129 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.738983 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b25b2d-cd54-4ac4-8f63-531946cef94e-tls-certs\") pod \"model-serving-api-54d89bd5fc-dlqj6\" (UID: \"28b25b2d-cd54-4ac4-8f63-531946cef94e\") " pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.809182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.809148 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-7jwk6\"" Jul 23 11:41:52.811219 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.811182 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:52.840073 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.840038 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxldr\" (UniqueName: \"kubernetes.io/projected/28b25b2d-cd54-4ac4-8f63-531946cef94e-kube-api-access-rxldr\") pod \"model-serving-api-54d89bd5fc-dlqj6\" (UID: \"28b25b2d-cd54-4ac4-8f63-531946cef94e\") " pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.840252 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.840116 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b25b2d-cd54-4ac4-8f63-531946cef94e-tls-certs\") pod \"model-serving-api-54d89bd5fc-dlqj6\" (UID: \"28b25b2d-cd54-4ac4-8f63-531946cef94e\") " pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.844022 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.843993 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28b25b2d-cd54-4ac4-8f63-531946cef94e-tls-certs\") pod \"model-serving-api-54d89bd5fc-dlqj6\" (UID: \"28b25b2d-cd54-4ac4-8f63-531946cef94e\") " pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.854899 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.854834 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxldr\" (UniqueName: \"kubernetes.io/projected/28b25b2d-cd54-4ac4-8f63-531946cef94e-kube-api-access-rxldr\") pod \"model-serving-api-54d89bd5fc-dlqj6\" (UID: \"28b25b2d-cd54-4ac4-8f63-531946cef94e\") " pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:52.998773 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.998748 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-9cwng"] Jul 23 11:41:52.999866 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:52.999835 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:53.001866 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:41:53.001839 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde982565_fe14_4b51_84fa_3ff6144a1617.slice/crio-2a68b43e7c1123eb23305669e718bdd4d6b9c64457bc1b0ad922eb4f93b4f708 WatchSource:0}: Error finding container 2a68b43e7c1123eb23305669e718bdd4d6b9c64457bc1b0ad922eb4f93b4f708: Status 404 returned error can't find the container with id 2a68b43e7c1123eb23305669e718bdd4d6b9c64457bc1b0ad922eb4f93b4f708 Jul 23 11:41:53.115577 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.115535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" event={"ID":"de982565-fe14-4b51-84fa-3ff6144a1617","Type":"ContainerStarted","Data":"2a68b43e7c1123eb23305669e718bdd4d6b9c64457bc1b0ad922eb4f93b4f708"} Jul 23 11:41:53.173547 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.173518 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dlqj6"] Jul 23 11:41:53.176059 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:41:53.176022 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b25b2d_cd54_4ac4_8f63_531946cef94e.slice/crio-1179507d25efa8d01b19470de90f5381578af6fb1b7346d972d09623f0f1f6f9 WatchSource:0}: Error finding container 1179507d25efa8d01b19470de90f5381578af6fb1b7346d972d09623f0f1f6f9: Status 404 returned error can't find the container with id 1179507d25efa8d01b19470de90f5381578af6fb1b7346d972d09623f0f1f6f9 Jul 23 11:41:53.638195 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.638154 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vpll6"] Jul 23 11:41:53.642975 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.642950 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.645870 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.645845 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 23 11:41:53.646194 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.646174 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-sfhtx\"" Jul 23 11:41:53.646374 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.646363 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 23 11:41:53.646652 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.646617 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 23 11:41:53.646893 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.646877 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 23 11:41:53.660193 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.660168 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vpll6"] Jul 23 11:41:53.749896 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.749863 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-istio-token\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.749896 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.749900 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-cacerts\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.750392 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.749934 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35a3665-7631-4767-b66d-e512d82af01b-local-certs\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.750392 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.750002 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2gvz\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-kube-api-access-l2gvz\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.750392 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.750059 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.750392 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.750094 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.750392 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.750227 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851089 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851054 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851115 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-istio-token\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851138 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-cacerts\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851168 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35a3665-7631-4767-b66d-e512d82af01b-local-certs\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851193 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2gvz\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-kube-api-access-l2gvz\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851222 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851300 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851253 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.851888 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.851773 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.854169 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.854143 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35a3665-7631-4767-b66d-e512d82af01b-local-certs\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.854323 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.854301 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.854727 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.854704 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-cacerts\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.855169 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.855143 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.867002 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.866947 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2gvz\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-kube-api-access-l2gvz\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.869222 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.869166 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-istio-token\") pod \"istiod-openshift-gateway-7574459656-vpll6\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:53.952950 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:53.952865 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:41:54.127269 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:54.127227 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" event={"ID":"28b25b2d-cd54-4ac4-8f63-531946cef94e","Type":"ContainerStarted","Data":"1179507d25efa8d01b19470de90f5381578af6fb1b7346d972d09623f0f1f6f9"} Jul 23 11:41:54.151859 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:54.151826 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vpll6"] Jul 23 11:41:54.155534 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:41:54.155499 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb35a3665_7631_4767_b66d_e512d82af01b.slice/crio-76d9567b3e92ce9acfb4af983b53a2cecd6b6ee98bfb62879f460a90841cbeb8 WatchSource:0}: Error finding container 76d9567b3e92ce9acfb4af983b53a2cecd6b6ee98bfb62879f460a90841cbeb8: Status 404 returned error can't find the container with id 76d9567b3e92ce9acfb4af983b53a2cecd6b6ee98bfb62879f460a90841cbeb8 Jul 23 11:41:55.133179 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:55.133142 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" event={"ID":"b35a3665-7631-4767-b66d-e512d82af01b","Type":"ContainerStarted","Data":"76d9567b3e92ce9acfb4af983b53a2cecd6b6ee98bfb62879f460a90841cbeb8"} Jul 23 11:41:56.822587 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:56.822475 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 23 11:41:57.145509 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.145414 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" event={"ID":"3cfd86c0-746c-46e9-a026-487b5d3f81ff","Type":"ContainerStarted","Data":"404e680aa021af6b3ed40e9ad8a984fb1a3b2993882990a03c683da7e2c8ac3f"} Jul 23 11:41:57.147021 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.146982 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" event={"ID":"28b25b2d-cd54-4ac4-8f63-531946cef94e","Type":"ContainerStarted","Data":"efaaa9f76e0ab9a9acd1f4b7bfa28737f7f33fa9e8789c161990b0b63ac8cb61"} Jul 23 11:41:57.147155 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.147066 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:41:57.148420 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.148397 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" event={"ID":"de982565-fe14-4b51-84fa-3ff6144a1617","Type":"ContainerStarted","Data":"18ca0a5153910861dcc9cf0745b15802a772f8f4d06ac35e7677abff749e79a6"} Jul 23 11:41:57.148553 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.148531 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:41:57.165523 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.165458 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b7f786dc8-zxjwr" podStartSLOduration=1.223459336 podStartE2EDuration="6.165435972s" totalImagesPullingTime="4.941976636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:57.163519345 +0000 UTC m=+365.268845759" watchObservedRunningTime="2026-07-23 11:41:57.165435972 +0000 UTC m=+365.270762383" Jul 23 11:41:57.185915 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:41:57.185843 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" podStartSLOduration=1.5402071670000002 podStartE2EDuration="5.185819597s" totalImagesPullingTime="3.64561243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:57.183402531 +0000 UTC m=+365.288728946" watchObservedRunningTime="2026-07-23 11:41:57.185819597 +0000 UTC m=+365.291146010" Jul 23 11:42:02.488264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.488211 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" podStartSLOduration=7.619310886 podStartE2EDuration="11.488193606s" totalImagesPullingTime="3.86888272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:41:57.199424802 +0000 UTC m=+365.304751215" watchObservedRunningTime="2026-07-23 11:42:02.488193606 +0000 UTC m=+370.593520018" Jul 23 11:42:02.489165 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.489140 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9"] Jul 23 11:42:02.492585 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.492564 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t2k79"] Jul 23 11:42:02.492709 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.492693 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.495870 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.495850 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.497353 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.497331 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-gzxjd\"" Jul 23 11:42:02.497901 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.497882 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 23 11:42:02.498058 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.498022 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 23 11:42:02.498758 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.498740 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-kbd6d\"" Jul 23 11:42:02.507671 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.507647 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9"] Jul 23 11:42:02.511997 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.511973 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t2k79"] Jul 23 11:42:02.638831 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.638781 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw7mm\" (UniqueName: \"kubernetes.io/projected/778eb9a6-8129-4808-948a-470d451f8304-kube-api-access-fw7mm\") pod \"kserve-controller-manager-d6f4b65b8-t2k79\" (UID: \"778eb9a6-8129-4808-948a-470d451f8304\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.639013 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.638850 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/778eb9a6-8129-4808-948a-470d451f8304-cert\") pod \"kserve-controller-manager-d6f4b65b8-t2k79\" (UID: \"778eb9a6-8129-4808-948a-470d451f8304\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.639013 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.638906 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3834800-cac3-4f1d-8b54-670f8f506e64-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wxvg9\" (UID: \"b3834800-cac3-4f1d-8b54-670f8f506e64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.639013 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.638929 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj824\" (UniqueName: \"kubernetes.io/projected/b3834800-cac3-4f1d-8b54-670f8f506e64-kube-api-access-pj824\") pod \"llmisvc-controller-manager-777c6bd7d5-wxvg9\" (UID: \"b3834800-cac3-4f1d-8b54-670f8f506e64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.740140 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.740035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3834800-cac3-4f1d-8b54-670f8f506e64-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wxvg9\" (UID: \"b3834800-cac3-4f1d-8b54-670f8f506e64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.740140 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.740087 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj824\" (UniqueName: \"kubernetes.io/projected/b3834800-cac3-4f1d-8b54-670f8f506e64-kube-api-access-pj824\") pod \"llmisvc-controller-manager-777c6bd7d5-wxvg9\" (UID: \"b3834800-cac3-4f1d-8b54-670f8f506e64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.740382 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.740158 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw7mm\" (UniqueName: \"kubernetes.io/projected/778eb9a6-8129-4808-948a-470d451f8304-kube-api-access-fw7mm\") pod \"kserve-controller-manager-d6f4b65b8-t2k79\" (UID: \"778eb9a6-8129-4808-948a-470d451f8304\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.740382 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.740219 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/778eb9a6-8129-4808-948a-470d451f8304-cert\") pod \"kserve-controller-manager-d6f4b65b8-t2k79\" (UID: \"778eb9a6-8129-4808-948a-470d451f8304\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.743010 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.742984 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b3834800-cac3-4f1d-8b54-670f8f506e64-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-wxvg9\" (UID: \"b3834800-cac3-4f1d-8b54-670f8f506e64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.743140 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.743016 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/778eb9a6-8129-4808-948a-470d451f8304-cert\") pod \"kserve-controller-manager-d6f4b65b8-t2k79\" (UID: \"778eb9a6-8129-4808-948a-470d451f8304\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.753527 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.753487 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw7mm\" (UniqueName: \"kubernetes.io/projected/778eb9a6-8129-4808-948a-470d451f8304-kube-api-access-fw7mm\") pod \"kserve-controller-manager-d6f4b65b8-t2k79\" (UID: \"778eb9a6-8129-4808-948a-470d451f8304\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.754653 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.754614 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj824\" (UniqueName: \"kubernetes.io/projected/b3834800-cac3-4f1d-8b54-670f8f506e64-kube-api-access-pj824\") pod \"llmisvc-controller-manager-777c6bd7d5-wxvg9\" (UID: \"b3834800-cac3-4f1d-8b54-670f8f506e64\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.804592 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.804547 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:02.810937 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.810914 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:02.990036 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:02.989994 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9"] Jul 23 11:42:02.992292 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:42:02.992233 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb3834800_cac3_4f1d_8b54_670f8f506e64.slice/crio-9927bb3c10cdabcd34851578eb1443183b39a28a3da3aa9a863e9118ac3e80ad WatchSource:0}: Error finding container 9927bb3c10cdabcd34851578eb1443183b39a28a3da3aa9a863e9118ac3e80ad: Status 404 returned error can't find the container with id 9927bb3c10cdabcd34851578eb1443183b39a28a3da3aa9a863e9118ac3e80ad Jul 23 11:42:03.010666 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:03.010641 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-t2k79"] Jul 23 11:42:03.013478 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:42:03.013451 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778eb9a6_8129_4808_948a_470d451f8304.slice/crio-28855828348f839ab941a9861742e1370aed18467628b5d725e28f732158a879 WatchSource:0}: Error finding container 28855828348f839ab941a9861742e1370aed18467628b5d725e28f732158a879: Status 404 returned error can't find the container with id 28855828348f839ab941a9861742e1370aed18467628b5d725e28f732158a879 Jul 23 11:42:03.181439 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:03.181403 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" event={"ID":"778eb9a6-8129-4808-948a-470d451f8304","Type":"ContainerStarted","Data":"28855828348f839ab941a9861742e1370aed18467628b5d725e28f732158a879"} Jul 23 11:42:03.182525 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:03.182499 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" event={"ID":"b3834800-cac3-4f1d-8b54-670f8f506e64","Type":"ContainerStarted","Data":"9927bb3c10cdabcd34851578eb1443183b39a28a3da3aa9a863e9118ac3e80ad"} Jul 23 11:42:05.143362 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.143314 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb"] Jul 23 11:42:05.146886 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.146865 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.157337 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.157312 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-dvtbs\"" Jul 23 11:42:05.186364 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.186328 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb"] Jul 23 11:42:05.236113 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.236079 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vpll6"] Jul 23 11:42:05.271822 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.271787 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.271991 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.271843 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.271991 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.271875 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/87931ee1-f888-472f-8be2-98c84ac77bd3-kube-api-access-9gjlq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.271991 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.271897 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.271991 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.271961 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.272144 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.271997 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.272144 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.272025 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87931ee1-f888-472f-8be2-98c84ac77bd3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.272144 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.272066 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.272144 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.272094 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373133 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373091 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373146 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373213 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373246 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/87931ee1-f888-472f-8be2-98c84ac77bd3-kube-api-access-9gjlq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373274 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373313 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373585 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373365 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373585 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373407 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87931ee1-f888-472f-8be2-98c84ac77bd3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373585 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373462 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373585 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373551 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373821 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373768 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373821 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373805 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.373931 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.373889 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.374315 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.374290 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/87931ee1-f888-472f-8be2-98c84ac77bd3-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.375978 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.375951 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.376160 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.376134 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.413949 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.413867 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gjlq\" (UniqueName: \"kubernetes.io/projected/87931ee1-f888-472f-8be2-98c84ac77bd3-kube-api-access-9gjlq\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.428319 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.428273 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/87931ee1-f888-472f-8be2-98c84ac77bd3-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4h2ncb\" (UID: \"87931ee1-f888-472f-8be2-98c84ac77bd3\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:05.459791 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:05.459751 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:06.027790 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.027702 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb"] Jul 23 11:42:06.029824 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:42:06.029792 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87931ee1_f888_472f_8be2_98c84ac77bd3.slice/crio-f17e62608260268a244a9f84a8d6a3847415647249edae2824a597d240ecc0ed WatchSource:0}: Error finding container f17e62608260268a244a9f84a8d6a3847415647249edae2824a597d240ecc0ed: Status 404 returned error can't find the container with id f17e62608260268a244a9f84a8d6a3847415647249edae2824a597d240ecc0ed Jul 23 11:42:06.205105 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.205018 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" event={"ID":"778eb9a6-8129-4808-948a-470d451f8304","Type":"ContainerStarted","Data":"da58686b2957296f2ec258251a4dac4633fcc70a9c018c806a596e3c44ce1ccf"} Jul 23 11:42:06.205559 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.205189 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:06.207186 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.207086 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" event={"ID":"b3834800-cac3-4f1d-8b54-670f8f506e64","Type":"ContainerStarted","Data":"2d117999353b8a21b677ffcd48dc3e287d4d29e182f52ef4181e351d7450985c"} Jul 23 11:42:06.207186 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.207142 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:06.208857 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.208830 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" event={"ID":"87931ee1-f888-472f-8be2-98c84ac77bd3","Type":"ContainerStarted","Data":"f17e62608260268a244a9f84a8d6a3847415647249edae2824a597d240ecc0ed"} Jul 23 11:42:06.229157 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:06.229087 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" podStartSLOduration=1.389086119 podStartE2EDuration="4.229067426s" totalImagesPullingTime="2.839981307s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:06.226228504 +0000 UTC m=+374.331554936" watchObservedRunningTime="2026-07-23 11:42:06.229067426 +0000 UTC m=+374.334393838" Jul 23 11:42:07.802074 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:07.802037 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:42:07.802322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:07.802109 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:42:08.155150 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.155070 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-9cwng" Jul 23 11:42:08.157422 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.157385 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dlqj6" Jul 23 11:42:08.174155 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.174093 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" podStartSLOduration=3.306628779 podStartE2EDuration="6.174071534s" totalImagesPullingTime="2.867442755s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:06.258515887 +0000 UTC m=+374.363842322" watchObservedRunningTime="2026-07-23 11:42:08.174071534 +0000 UTC m=+376.279397945" Jul 23 11:42:08.224571 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.224338 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" event={"ID":"b35a3665-7631-4767-b66d-e512d82af01b","Type":"ContainerStarted","Data":"49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f"} Jul 23 11:42:08.224571 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.224404 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" podUID="b35a3665-7631-4767-b66d-e512d82af01b" containerName="discovery" containerID="cri-o://49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f" gracePeriod=30 Jul 23 11:42:08.224828 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.224782 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:42:08.226340 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.226310 2554 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-vpll6 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 11:42:08.226450 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.226370 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" podUID="b35a3665-7631-4767-b66d-e512d82af01b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 11:42:08.251154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.251097 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" podStartSLOduration=1.609599419 podStartE2EDuration="15.251074239s" totalImagesPullingTime="13.64147482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:41:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:08.248607951 +0000 UTC m=+376.353934399" watchObservedRunningTime="2026-07-23 11:42:08.251074239 +0000 UTC m=+376.356400653" Jul 23 11:42:08.508180 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.508150 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:42:08.605695 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605657 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35a3665-7631-4767-b66d-e512d82af01b-local-certs\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.605872 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605718 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-dns-cert\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.605872 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605750 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-ca-configmap\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.605872 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605828 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2gvz\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-kube-api-access-l2gvz\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.606027 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605879 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-istio-token\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.606027 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605916 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-kubeconfig\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.606027 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.605972 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-cacerts\") pod \"b35a3665-7631-4767-b66d-e512d82af01b\" (UID: \"b35a3665-7631-4767-b66d-e512d82af01b\") " Jul 23 11:42:08.606441 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.606390 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:42:08.609040 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.609010 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-cacerts" (OuterVolumeSpecName: "cacerts") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:42:08.609168 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.609033 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:42:08.609168 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.609143 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-kube-api-access-l2gvz" (OuterVolumeSpecName: "kube-api-access-l2gvz") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "kube-api-access-l2gvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:42:08.609322 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.609305 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b35a3665-7631-4767-b66d-e512d82af01b-local-certs" (OuterVolumeSpecName: "local-certs") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:42:08.609857 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.609833 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:42:08.609972 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.609946 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b35a3665-7631-4767-b66d-e512d82af01b" (UID: "b35a3665-7631-4767-b66d-e512d82af01b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:42:08.706776 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706696 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-dns-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:08.706776 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706732 2554 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b35a3665-7631-4767-b66d-e512d82af01b-istio-csr-ca-configmap\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:08.706776 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706748 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2gvz\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-kube-api-access-l2gvz\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:08.706776 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706763 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b35a3665-7631-4767-b66d-e512d82af01b-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:08.706776 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706778 2554 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-istio-kubeconfig\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:08.707122 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706799 2554 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b35a3665-7631-4767-b66d-e512d82af01b-cacerts\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:08.707122 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:08.706814 2554 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b35a3665-7631-4767-b66d-e512d82af01b-local-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:09.231412 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.231371 2554 generic.go:356] "Generic (PLEG): container finished" podID="b35a3665-7631-4767-b66d-e512d82af01b" containerID="49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f" exitCode=2 Jul 23 11:42:09.231863 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.231423 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" Jul 23 11:42:09.231863 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.231423 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" event={"ID":"b35a3665-7631-4767-b66d-e512d82af01b","Type":"ContainerDied","Data":"49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f"} Jul 23 11:42:09.231863 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.231487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-vpll6" event={"ID":"b35a3665-7631-4767-b66d-e512d82af01b","Type":"ContainerDied","Data":"76d9567b3e92ce9acfb4af983b53a2cecd6b6ee98bfb62879f460a90841cbeb8"} Jul 23 11:42:09.231863 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.231519 2554 scope.go:117] "RemoveContainer" containerID="49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f" Jul 23 11:42:09.255261 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.255229 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vpll6"] Jul 23 11:42:09.260749 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.260714 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-vpll6"] Jul 23 11:42:09.824873 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.824848 2554 scope.go:117] "RemoveContainer" containerID="49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f" Jul 23 11:42:09.825219 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:42:09.825200 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f\": container with ID starting with 49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f not found: ID does not exist" containerID="49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f" Jul 23 11:42:09.825295 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.825230 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f"} err="failed to get container status \"49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f\": rpc error: code = NotFound desc = could not find container \"49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f\": container with ID starting with 49e1a359c27dd831d41f5a14da16d100de04e50f6bb77ddbb74ed64491a3dd6f not found: ID does not exist" Jul 23 11:42:09.890389 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.890359 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:42:09.890491 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.890433 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:42:09.890491 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:09.890462 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:42:10.237537 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:10.237436 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" event={"ID":"87931ee1-f888-472f-8be2-98c84ac77bd3","Type":"ContainerStarted","Data":"d351b220e75d7a90ad3f2a28298df97d59df7409700f1aae493a05db604d1ff3"} Jul 23 11:42:10.263142 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:10.263088 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" podStartSLOduration=1.406824029 podStartE2EDuration="5.263068529s" totalImagesPullingTime="3.8562445s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:10.261312096 +0000 UTC m=+378.366638535" watchObservedRunningTime="2026-07-23 11:42:10.263068529 +0000 UTC m=+378.368394940" Jul 23 11:42:10.452966 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:10.452932 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b35a3665-7631-4767-b66d-e512d82af01b" path="/var/lib/kubelet/pods/b35a3665-7631-4767-b66d-e512d82af01b/volumes" Jul 23 11:42:10.459959 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:10.459932 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:10.464904 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:10.464883 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:11.243192 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:11.243159 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:11.244387 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:11.244367 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4h2ncb" Jul 23 11:42:37.216378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:37.216343 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-wxvg9" Jul 23 11:42:37.219656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:37.219614 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-t2k79" Jul 23 11:42:52.503714 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.503673 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4n9"] Jul 23 11:42:52.504103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.504052 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b35a3665-7631-4767-b66d-e512d82af01b" containerName="discovery" Jul 23 11:42:52.504103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.504064 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b35a3665-7631-4767-b66d-e512d82af01b" containerName="discovery" Jul 23 11:42:52.504175 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.504126 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b35a3665-7631-4767-b66d-e512d82af01b" containerName="discovery" Jul 23 11:42:52.508317 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.508301 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:52.513348 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.513329 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 23 11:42:52.513484 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.513384 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 23 11:42:52.514029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.514007 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-ktxn7\"" Jul 23 11:42:52.529686 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.529657 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4n9"] Jul 23 11:42:52.602529 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.602435 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnthg\" (UniqueName: \"kubernetes.io/projected/6707065f-3182-4409-b8dd-caf83bd89f41-kube-api-access-qnthg\") pod \"kuadrant-operator-catalog-wl4n9\" (UID: \"6707065f-3182-4409-b8dd-caf83bd89f41\") " pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:52.703489 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.703451 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnthg\" (UniqueName: \"kubernetes.io/projected/6707065f-3182-4409-b8dd-caf83bd89f41-kube-api-access-qnthg\") pod \"kuadrant-operator-catalog-wl4n9\" (UID: \"6707065f-3182-4409-b8dd-caf83bd89f41\") " pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:52.711966 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.711944 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnthg\" (UniqueName: \"kubernetes.io/projected/6707065f-3182-4409-b8dd-caf83bd89f41-kube-api-access-qnthg\") pod \"kuadrant-operator-catalog-wl4n9\" (UID: \"6707065f-3182-4409-b8dd-caf83bd89f41\") " pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:52.818266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.818237 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:52.856611 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.856525 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4n9"] Jul 23 11:42:52.956885 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:52.956751 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4n9"] Jul 23 11:42:52.959759 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:42:52.959730 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6707065f_3182_4409_b8dd_caf83bd89f41.slice/crio-342ac06838b36739b52e75c6ff4f1a1fb899604a51182b3f306fffe15b949b41 WatchSource:0}: Error finding container 342ac06838b36739b52e75c6ff4f1a1fb899604a51182b3f306fffe15b949b41: Status 404 returned error can't find the container with id 342ac06838b36739b52e75c6ff4f1a1fb899604a51182b3f306fffe15b949b41 Jul 23 11:42:53.066702 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.066666 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xdwxf"] Jul 23 11:42:53.069487 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.069470 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:42:53.079145 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.079114 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xdwxf"] Jul 23 11:42:53.209577 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.209485 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvv2x\" (UniqueName: \"kubernetes.io/projected/a3ebb31f-0f6c-465d-b304-cff9297e8865-kube-api-access-tvv2x\") pod \"kuadrant-operator-catalog-xdwxf\" (UID: \"a3ebb31f-0f6c-465d-b304-cff9297e8865\") " pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:42:53.309961 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.309928 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvv2x\" (UniqueName: \"kubernetes.io/projected/a3ebb31f-0f6c-465d-b304-cff9297e8865-kube-api-access-tvv2x\") pod \"kuadrant-operator-catalog-xdwxf\" (UID: \"a3ebb31f-0f6c-465d-b304-cff9297e8865\") " pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:42:53.319143 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.319114 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvv2x\" (UniqueName: \"kubernetes.io/projected/a3ebb31f-0f6c-465d-b304-cff9297e8865-kube-api-access-tvv2x\") pod \"kuadrant-operator-catalog-xdwxf\" (UID: \"a3ebb31f-0f6c-465d-b304-cff9297e8865\") " pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:42:53.379672 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.379607 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:42:53.439889 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.439859 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" event={"ID":"6707065f-3182-4409-b8dd-caf83bd89f41","Type":"ContainerStarted","Data":"342ac06838b36739b52e75c6ff4f1a1fb899604a51182b3f306fffe15b949b41"} Jul 23 11:42:53.517584 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:53.517482 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-xdwxf"] Jul 23 11:42:53.520103 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:42:53.520077 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3ebb31f_0f6c_465d_b304_cff9297e8865.slice/crio-50cee44409009e95cb1b3fda26f4905cb9a8e94cc2bcf86f92f19db42fadb1e3 WatchSource:0}: Error finding container 50cee44409009e95cb1b3fda26f4905cb9a8e94cc2bcf86f92f19db42fadb1e3: Status 404 returned error can't find the container with id 50cee44409009e95cb1b3fda26f4905cb9a8e94cc2bcf86f92f19db42fadb1e3 Jul 23 11:42:54.452180 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:54.452144 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" event={"ID":"a3ebb31f-0f6c-465d-b304-cff9297e8865","Type":"ContainerStarted","Data":"50cee44409009e95cb1b3fda26f4905cb9a8e94cc2bcf86f92f19db42fadb1e3"} Jul 23 11:42:55.455768 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.455726 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" event={"ID":"a3ebb31f-0f6c-465d-b304-cff9297e8865","Type":"ContainerStarted","Data":"bf22a2252cc6ea53e6a0bfb5a66f8bda5cf6f99a5a70a41163f5c30c67e89d74"} Jul 23 11:42:55.457392 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.457369 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" event={"ID":"6707065f-3182-4409-b8dd-caf83bd89f41","Type":"ContainerStarted","Data":"f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f"} Jul 23 11:42:55.457509 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.457423 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" podUID="6707065f-3182-4409-b8dd-caf83bd89f41" containerName="registry-server" containerID="cri-o://f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f" gracePeriod=2 Jul 23 11:42:55.473459 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.473408 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" podStartSLOduration=0.99197671 podStartE2EDuration="2.473390025s" totalImagesPullingTime="1.481413315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:55.472014099 +0000 UTC m=+423.577340536" watchObservedRunningTime="2026-07-23 11:42:55.473390025 +0000 UTC m=+423.578716462" Jul 23 11:42:55.489420 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.489363 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" podStartSLOduration=1.44859697 podStartE2EDuration="3.48933966s" totalImagesPullingTime="2.04074269s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:42:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:42:55.488276599 +0000 UTC m=+423.593603010" watchObservedRunningTime="2026-07-23 11:42:55.48933966 +0000 UTC m=+423.594666073" Jul 23 11:42:55.699379 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.699348 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:55.834613 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.834580 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qnthg\" (UniqueName: \"kubernetes.io/projected/6707065f-3182-4409-b8dd-caf83bd89f41-kube-api-access-qnthg\") pod \"6707065f-3182-4409-b8dd-caf83bd89f41\" (UID: \"6707065f-3182-4409-b8dd-caf83bd89f41\") " Jul 23 11:42:55.836838 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.836817 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6707065f-3182-4409-b8dd-caf83bd89f41-kube-api-access-qnthg" (OuterVolumeSpecName: "kube-api-access-qnthg") pod "6707065f-3182-4409-b8dd-caf83bd89f41" (UID: "6707065f-3182-4409-b8dd-caf83bd89f41"). InnerVolumeSpecName "kube-api-access-qnthg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:42:55.935541 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:55.935487 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnthg\" (UniqueName: \"kubernetes.io/projected/6707065f-3182-4409-b8dd-caf83bd89f41-kube-api-access-qnthg\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:42:56.463249 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.463220 2554 generic.go:356] "Generic (PLEG): container finished" podID="6707065f-3182-4409-b8dd-caf83bd89f41" containerID="f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f" exitCode=0 Jul 23 11:42:56.463551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.463263 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" Jul 23 11:42:56.463551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.463299 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" event={"ID":"6707065f-3182-4409-b8dd-caf83bd89f41","Type":"ContainerDied","Data":"f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f"} Jul 23 11:42:56.463551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.463349 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-wl4n9" event={"ID":"6707065f-3182-4409-b8dd-caf83bd89f41","Type":"ContainerDied","Data":"342ac06838b36739b52e75c6ff4f1a1fb899604a51182b3f306fffe15b949b41"} Jul 23 11:42:56.463551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.463374 2554 scope.go:117] "RemoveContainer" containerID="f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f" Jul 23 11:42:56.472237 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.472221 2554 scope.go:117] "RemoveContainer" containerID="f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f" Jul 23 11:42:56.472493 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:42:56.472476 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f\": container with ID starting with f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f not found: ID does not exist" containerID="f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f" Jul 23 11:42:56.472535 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.472501 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f"} err="failed to get container status \"f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f\": rpc error: code = NotFound desc = could not find container \"f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f\": container with ID starting with f2e90ec690cf4bbe8518b97804a3cec2bff487ca68278844ace68eeab6707c1f not found: ID does not exist" Jul 23 11:42:56.479024 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.478999 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4n9"] Jul 23 11:42:56.485025 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:56.485005 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-wl4n9"] Jul 23 11:42:58.453335 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:42:58.453298 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6707065f-3182-4409-b8dd-caf83bd89f41" path="/var/lib/kubelet/pods/6707065f-3182-4409-b8dd-caf83bd89f41/volumes" Jul 23 11:43:03.380070 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:03.380033 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:43:03.380489 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:03.380094 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:43:03.402805 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:03.402771 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:43:03.519397 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:03.519371 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-xdwxf" Jul 23 11:43:25.417412 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.417316 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-cb862"] Jul 23 11:43:25.418044 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.417902 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6707065f-3182-4409-b8dd-caf83bd89f41" containerName="registry-server" Jul 23 11:43:25.418044 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.417925 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="6707065f-3182-4409-b8dd-caf83bd89f41" containerName="registry-server" Jul 23 11:43:25.418044 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.418029 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="6707065f-3182-4409-b8dd-caf83bd89f41" containerName="registry-server" Jul 23 11:43:25.426259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.426215 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:25.430485 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.430456 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-kzwb5\"" Jul 23 11:43:25.492260 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.492221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ftf9\" (UniqueName: \"kubernetes.io/projected/50fe6d81-eab3-4d54-9f7b-65fe0272b4b2-kube-api-access-5ftf9\") pod \"authorino-operator-75cc8d5cb7-cb862\" (UID: \"50fe6d81-eab3-4d54-9f7b-65fe0272b4b2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:25.522052 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.522019 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-cb862"] Jul 23 11:43:25.593259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.593218 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ftf9\" (UniqueName: \"kubernetes.io/projected/50fe6d81-eab3-4d54-9f7b-65fe0272b4b2-kube-api-access-5ftf9\") pod \"authorino-operator-75cc8d5cb7-cb862\" (UID: \"50fe6d81-eab3-4d54-9f7b-65fe0272b4b2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:25.604274 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.604237 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ftf9\" (UniqueName: \"kubernetes.io/projected/50fe6d81-eab3-4d54-9f7b-65fe0272b4b2-kube-api-access-5ftf9\") pod \"authorino-operator-75cc8d5cb7-cb862\" (UID: \"50fe6d81-eab3-4d54-9f7b-65fe0272b4b2\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:25.737197 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.737106 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:25.900231 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:25.900203 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-cb862"] Jul 23 11:43:25.903211 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:25.903185 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50fe6d81_eab3_4d54_9f7b_65fe0272b4b2.slice/crio-a72412ff2a2ff405df055c10f09d1602c14538e196c01c9f21aa4fc291013ba0 WatchSource:0}: Error finding container a72412ff2a2ff405df055c10f09d1602c14538e196c01c9f21aa4fc291013ba0: Status 404 returned error can't find the container with id a72412ff2a2ff405df055c10f09d1602c14538e196c01c9f21aa4fc291013ba0 Jul 23 11:43:26.603445 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:26.603408 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" event={"ID":"50fe6d81-eab3-4d54-9f7b-65fe0272b4b2","Type":"ContainerStarted","Data":"a72412ff2a2ff405df055c10f09d1602c14538e196c01c9f21aa4fc291013ba0"} Jul 23 11:43:28.666128 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.666082 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz"] Jul 23 11:43:28.670224 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.670201 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:28.673275 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.673250 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-hccgv\"" Jul 23 11:43:28.690141 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.690111 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz"] Jul 23 11:43:28.724755 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.724719 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhfj\" (UniqueName: \"kubernetes.io/projected/f1efdee5-e0b9-44c6-a001-566063caebca-kube-api-access-2fhfj\") pod \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" (UID: \"f1efdee5-e0b9-44c6-a001-566063caebca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:28.825536 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.825490 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhfj\" (UniqueName: \"kubernetes.io/projected/f1efdee5-e0b9-44c6-a001-566063caebca-kube-api-access-2fhfj\") pod \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" (UID: \"f1efdee5-e0b9-44c6-a001-566063caebca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:28.836047 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.836016 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhfj\" (UniqueName: \"kubernetes.io/projected/f1efdee5-e0b9-44c6-a001-566063caebca-kube-api-access-2fhfj\") pod \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" (UID: \"f1efdee5-e0b9-44c6-a001-566063caebca\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:28.984440 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:28.984396 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:29.136399 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:29.136362 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz"] Jul 23 11:43:29.139662 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:29.139611 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1efdee5_e0b9_44c6_a001_566063caebca.slice/crio-9a99880d4d2ae7e9b266cc888e27560db1be68541582f4293fedf92cc61b4b63 WatchSource:0}: Error finding container 9a99880d4d2ae7e9b266cc888e27560db1be68541582f4293fedf92cc61b4b63: Status 404 returned error can't find the container with id 9a99880d4d2ae7e9b266cc888e27560db1be68541582f4293fedf92cc61b4b63 Jul 23 11:43:29.621739 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:29.621703 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" event={"ID":"f1efdee5-e0b9-44c6-a001-566063caebca","Type":"ContainerStarted","Data":"9a99880d4d2ae7e9b266cc888e27560db1be68541582f4293fedf92cc61b4b63"} Jul 23 11:43:29.623217 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:29.623190 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" event={"ID":"50fe6d81-eab3-4d54-9f7b-65fe0272b4b2","Type":"ContainerStarted","Data":"d7e418e540cb56956a34a657e94c5fc82f8ac8af06be0c2c428900f2b090c4c1"} Jul 23 11:43:29.623336 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:29.623298 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:29.641510 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:29.641457 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" podStartSLOduration=1.640731111 podStartE2EDuration="4.641437179s" totalImagesPullingTime="3.000706068s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:29.641381911 +0000 UTC m=+457.746708324" watchObservedRunningTime="2026-07-23 11:43:29.641437179 +0000 UTC m=+457.746763593" Jul 23 11:43:31.081976 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.081886 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd"] Jul 23 11:43:31.085314 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.085275 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:31.088267 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.088240 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-2hqt2\"" Jul 23 11:43:31.088406 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.088300 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 23 11:43:31.100041 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.100017 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd"] Jul 23 11:43:31.149285 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.149252 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-977m8\" (UniqueName: \"kubernetes.io/projected/444e9960-300a-4ee8-9a5d-464a343316cf-kube-api-access-977m8\") pod \"dns-operator-controller-manager-864dd78c4c-5stpd\" (UID: \"444e9960-300a-4ee8-9a5d-464a343316cf\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:31.250267 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.250225 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-977m8\" (UniqueName: \"kubernetes.io/projected/444e9960-300a-4ee8-9a5d-464a343316cf-kube-api-access-977m8\") pod \"dns-operator-controller-manager-864dd78c4c-5stpd\" (UID: \"444e9960-300a-4ee8-9a5d-464a343316cf\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:31.272161 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.272128 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-977m8\" (UniqueName: \"kubernetes.io/projected/444e9960-300a-4ee8-9a5d-464a343316cf-kube-api-access-977m8\") pod \"dns-operator-controller-manager-864dd78c4c-5stpd\" (UID: \"444e9960-300a-4ee8-9a5d-464a343316cf\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:31.396273 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.396168 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:31.535577 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.535553 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd"] Jul 23 11:43:31.538276 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:31.538247 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod444e9960_300a_4ee8_9a5d_464a343316cf.slice/crio-2a42cd4ac2bdd6b8509837836610c16164f66d1559defa87d2741c8415f96b6c WatchSource:0}: Error finding container 2a42cd4ac2bdd6b8509837836610c16164f66d1559defa87d2741c8415f96b6c: Status 404 returned error can't find the container with id 2a42cd4ac2bdd6b8509837836610c16164f66d1559defa87d2741c8415f96b6c Jul 23 11:43:31.638291 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.638255 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" event={"ID":"f1efdee5-e0b9-44c6-a001-566063caebca","Type":"ContainerStarted","Data":"dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02"} Jul 23 11:43:31.638472 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.638331 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:31.639478 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.639448 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" event={"ID":"444e9960-300a-4ee8-9a5d-464a343316cf","Type":"ContainerStarted","Data":"2a42cd4ac2bdd6b8509837836610c16164f66d1559defa87d2741c8415f96b6c"} Jul 23 11:43:31.663847 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:31.663737 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" podStartSLOduration=2.116330553 podStartE2EDuration="3.663716666s" totalImagesPullingTime="1.547386113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:31.663071648 +0000 UTC m=+459.768398059" watchObservedRunningTime="2026-07-23 11:43:31.663716666 +0000 UTC m=+459.769043081" Jul 23 11:43:32.965738 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:32.956612 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6849845f9-7dhj2"] Jul 23 11:43:32.965738 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:32.960139 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:32.977174 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:32.977148 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6849845f9-7dhj2"] Jul 23 11:43:33.067707 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.067671 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/323d76e9-03ed-428b-bffb-09a24eb84243-console-oauth-config\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.067896 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.067818 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-service-ca\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.067956 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.067905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-trusted-ca-bundle\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.067956 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.067932 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-oauth-serving-cert\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.067956 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.067947 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-console-config\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.068077 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.068006 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/323d76e9-03ed-428b-bffb-09a24eb84243-console-serving-cert\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.068077 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.068029 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xf4w\" (UniqueName: \"kubernetes.io/projected/323d76e9-03ed-428b-bffb-09a24eb84243-kube-api-access-2xf4w\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169003 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.168953 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-service-ca\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169041 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-trusted-ca-bundle\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169067 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-oauth-serving-cert\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-console-config\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169129 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/323d76e9-03ed-428b-bffb-09a24eb84243-console-serving-cert\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169155 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xf4w\" (UniqueName: \"kubernetes.io/projected/323d76e9-03ed-428b-bffb-09a24eb84243-kube-api-access-2xf4w\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169185 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/323d76e9-03ed-428b-bffb-09a24eb84243-console-oauth-config\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.169907 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169876 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-service-ca\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.170182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169876 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-oauth-serving-cert\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.170182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169961 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-console-config\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.170182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.169972 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/323d76e9-03ed-428b-bffb-09a24eb84243-trusted-ca-bundle\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.172018 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.171994 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/323d76e9-03ed-428b-bffb-09a24eb84243-console-serving-cert\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.172110 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.171996 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/323d76e9-03ed-428b-bffb-09a24eb84243-console-oauth-config\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.180440 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.180399 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xf4w\" (UniqueName: \"kubernetes.io/projected/323d76e9-03ed-428b-bffb-09a24eb84243-kube-api-access-2xf4w\") pod \"console-6849845f9-7dhj2\" (UID: \"323d76e9-03ed-428b-bffb-09a24eb84243\") " pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.273775 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.273688 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:33.807767 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:33.807729 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6849845f9-7dhj2"] Jul 23 11:43:33.816243 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:33.816210 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323d76e9_03ed_428b_bffb_09a24eb84243.slice/crio-e82593a89f9a0f1358f38447dc03ea9681536a240c85837f44f24b4c249b7ee6 WatchSource:0}: Error finding container e82593a89f9a0f1358f38447dc03ea9681536a240c85837f44f24b4c249b7ee6: Status 404 returned error can't find the container with id e82593a89f9a0f1358f38447dc03ea9681536a240c85837f44f24b4c249b7ee6 Jul 23 11:43:34.656595 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:34.656554 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6849845f9-7dhj2" event={"ID":"323d76e9-03ed-428b-bffb-09a24eb84243","Type":"ContainerStarted","Data":"68b6899664b4ca65aec56e69d9fd7e28c66528f7c7af565e3f593edf6042ca7a"} Jul 23 11:43:34.657092 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:34.656614 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6849845f9-7dhj2" event={"ID":"323d76e9-03ed-428b-bffb-09a24eb84243","Type":"ContainerStarted","Data":"e82593a89f9a0f1358f38447dc03ea9681536a240c85837f44f24b4c249b7ee6"} Jul 23 11:43:34.658045 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:34.658021 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" event={"ID":"444e9960-300a-4ee8-9a5d-464a343316cf","Type":"ContainerStarted","Data":"eca8d62b82385d815e537d1de5b5827d517d2c4c3da08987f65392c204ccc000"} Jul 23 11:43:34.658175 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:34.658160 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:34.685883 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:34.685836 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6849845f9-7dhj2" podStartSLOduration=2.685816863 podStartE2EDuration="2.685816863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:34.682444965 +0000 UTC m=+462.787771397" watchObservedRunningTime="2026-07-23 11:43:34.685816863 +0000 UTC m=+462.791143279" Jul 23 11:43:34.703902 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:34.703848 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" podStartSLOduration=1.568369389 podStartE2EDuration="3.703829389s" totalImagesPullingTime="2.13546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:34.702102124 +0000 UTC m=+462.807428562" watchObservedRunningTime="2026-07-23 11:43:34.703829389 +0000 UTC m=+462.809156102" Jul 23 11:43:35.475249 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.475212 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2"] Jul 23 11:43:35.479029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.479004 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.482416 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.482390 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qddrk\"" Jul 23 11:43:35.508524 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.508489 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2"] Jul 23 11:43:35.594010 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.593960 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c54x7\" (UniqueName: \"kubernetes.io/projected/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-kube-api-access-c54x7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.594010 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.594010 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.695150 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.695107 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c54x7\" (UniqueName: \"kubernetes.io/projected/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-kube-api-access-c54x7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.695150 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.695163 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.695798 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.695607 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.708052 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.708028 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c54x7\" (UniqueName: \"kubernetes.io/projected/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-kube-api-access-c54x7\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.790334 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.790301 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:35.938373 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:35.938344 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2"] Jul 23 11:43:35.940910 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:35.940883 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ca4d0e5_ade9_4ab2_b573_ef833907e0e1.slice/crio-174c956ea794311333e4b2a3f42007efd6009e2dc328a711921b19663fe97414 WatchSource:0}: Error finding container 174c956ea794311333e4b2a3f42007efd6009e2dc328a711921b19663fe97414: Status 404 returned error can't find the container with id 174c956ea794311333e4b2a3f42007efd6009e2dc328a711921b19663fe97414 Jul 23 11:43:36.669127 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:36.669093 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" event={"ID":"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1","Type":"ContainerStarted","Data":"174c956ea794311333e4b2a3f42007efd6009e2dc328a711921b19663fe97414"} Jul 23 11:43:40.633788 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:40.633756 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-cb862" Jul 23 11:43:40.695045 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:40.695008 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" event={"ID":"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1","Type":"ContainerStarted","Data":"7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48"} Jul 23 11:43:40.695213 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:40.695096 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:40.715803 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:40.715748 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" podStartSLOduration=1.407782785 podStartE2EDuration="5.715732445s" totalImagesPullingTime="4.30794966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:40.713148665 +0000 UTC m=+468.818475078" watchObservedRunningTime="2026-07-23 11:43:40.715732445 +0000 UTC m=+468.821058857" Jul 23 11:43:42.647956 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:42.647919 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:43.274219 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:43.274167 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:43.274404 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:43.274263 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:43.279298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:43.279275 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:43.714395 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:43.714346 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6849845f9-7dhj2" Jul 23 11:43:43.763744 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:43.763711 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7975bbc757-ldhlz"] Jul 23 11:43:45.666074 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:45.666039 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-5stpd" Jul 23 11:43:51.701709 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:51.701677 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:52.985954 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:52.985917 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g"] Jul 23 11:43:52.991037 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:52.991015 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.003894 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.003860 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g"] Jul 23 11:43:53.052048 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.052007 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dxl2\" (UniqueName: \"kubernetes.io/projected/a70456c7-2e83-4632-bd55-377ff4f0f1fa-kube-api-access-2dxl2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.052048 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.052052 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70456c7-2e83-4632-bd55-377ff4f0f1fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.109520 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.109480 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g"] Jul 23 11:43:53.109836 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.109815 2554 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-2dxl2], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" Jul 23 11:43:53.153081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.153044 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dxl2\" (UniqueName: \"kubernetes.io/projected/a70456c7-2e83-4632-bd55-377ff4f0f1fa-kube-api-access-2dxl2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.153081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.153086 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70456c7-2e83-4632-bd55-377ff4f0f1fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.153462 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.153445 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70456c7-2e83-4632-bd55-377ff4f0f1fa-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.167138 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.167103 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dxl2\" (UniqueName: \"kubernetes.io/projected/a70456c7-2e83-4632-bd55-377ff4f0f1fa-kube-api-access-2dxl2\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.634466 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.634425 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g"] Jul 23 11:43:53.638568 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.638535 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g"] Jul 23 11:43:53.659466 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.659433 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2"] Jul 23 11:43:53.659710 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.659669 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" containerName="manager" containerID="cri-o://7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48" gracePeriod=2 Jul 23 11:43:53.667925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.667870 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2"] Jul 23 11:43:53.670552 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.670517 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.697004 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.696966 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz"] Jul 23 11:43:53.697479 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.697424 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" containerName="manager" containerID="cri-o://dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02" gracePeriod=2 Jul 23 11:43:53.701313 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.701152 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.704225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.704138 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz"] Jul 23 11:43:53.704349 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.704237 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.734574 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.734538 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr"] Jul 23 11:43:53.735276 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.735239 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" containerName="manager" Jul 23 11:43:53.735276 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.735266 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" containerName="manager" Jul 23 11:43:53.735459 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.735300 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" containerName="manager" Jul 23 11:43:53.735459 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.735309 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" containerName="manager" Jul 23 11:43:53.735459 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.735401 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" containerName="manager" Jul 23 11:43:53.735459 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.735421 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" containerName="manager" Jul 23 11:43:53.739081 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.739064 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.743069 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.743039 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.747301 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.747277 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq"] Jul 23 11:43:53.750672 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.750610 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:43:53.753296 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.753273 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr"] Jul 23 11:43:53.758205 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.758182 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfxq\" (UniqueName: \"kubernetes.io/projected/9490542d-4d00-4e2e-83a7-294aba9ab51b-kube-api-access-gcfxq\") pod \"limitador-operator-controller-manager-859d7b78bd-6t9fq\" (UID: \"9490542d-4d00-4e2e-83a7-294aba9ab51b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:43:53.758298 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.758254 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx9tf\" (UniqueName: \"kubernetes.io/projected/80cd0f30-ec4e-4524-bf78-844b956f98a2-kube-api-access-fx9tf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-skvzr\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.758370 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.758330 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd0f30-ec4e-4524-bf78-844b956f98a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-skvzr\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.761105 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.761081 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.770673 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.770611 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq"] Jul 23 11:43:53.780241 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.780110 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.783887 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.783857 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.808560 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.808519 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.810680 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.810655 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.829863 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.829839 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.832455 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.832357 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.834371 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.834336 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.836416 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.836382 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.859044 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859012 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dxl2\" (UniqueName: \"kubernetes.io/projected/a70456c7-2e83-4632-bd55-377ff4f0f1fa-kube-api-access-2dxl2\") pod \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " Jul 23 11:43:53.859189 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859171 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70456c7-2e83-4632-bd55-377ff4f0f1fa-extensions-socket-volume\") pod \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\" (UID: \"a70456c7-2e83-4632-bd55-377ff4f0f1fa\") " Jul 23 11:43:53.859341 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859321 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx9tf\" (UniqueName: \"kubernetes.io/projected/80cd0f30-ec4e-4524-bf78-844b956f98a2-kube-api-access-fx9tf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-skvzr\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.859400 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859370 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd0f30-ec4e-4524-bf78-844b956f98a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-skvzr\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.859451 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859399 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a70456c7-2e83-4632-bd55-377ff4f0f1fa-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a70456c7-2e83-4632-bd55-377ff4f0f1fa" (UID: "a70456c7-2e83-4632-bd55-377ff4f0f1fa"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:53.859505 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859461 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfxq\" (UniqueName: \"kubernetes.io/projected/9490542d-4d00-4e2e-83a7-294aba9ab51b-kube-api-access-gcfxq\") pod \"limitador-operator-controller-manager-859d7b78bd-6t9fq\" (UID: \"9490542d-4d00-4e2e-83a7-294aba9ab51b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:43:53.859558 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859507 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a70456c7-2e83-4632-bd55-377ff4f0f1fa-extensions-socket-volume\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:43:53.859829 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.859756 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd0f30-ec4e-4524-bf78-844b956f98a2-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-skvzr\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.861225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.861198 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a70456c7-2e83-4632-bd55-377ff4f0f1fa-kube-api-access-2dxl2" (OuterVolumeSpecName: "kube-api-access-2dxl2") pod "a70456c7-2e83-4632-bd55-377ff4f0f1fa" (UID: "a70456c7-2e83-4632-bd55-377ff4f0f1fa"). InnerVolumeSpecName "kube-api-access-2dxl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:53.879452 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.879390 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfxq\" (UniqueName: \"kubernetes.io/projected/9490542d-4d00-4e2e-83a7-294aba9ab51b-kube-api-access-gcfxq\") pod \"limitador-operator-controller-manager-859d7b78bd-6t9fq\" (UID: \"9490542d-4d00-4e2e-83a7-294aba9ab51b\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:43:53.879796 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.879773 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx9tf\" (UniqueName: \"kubernetes.io/projected/80cd0f30-ec4e-4524-bf78-844b956f98a2-kube-api-access-fx9tf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-skvzr\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:53.943453 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.943425 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.946033 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.946000 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.946778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.946762 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.948599 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.948578 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.952515 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.952490 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:53.955197 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.955169 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-pq2qz\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:53.957069 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.957046 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6d6v2\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:53.960417 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.960387 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c54x7\" (UniqueName: \"kubernetes.io/projected/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-kube-api-access-c54x7\") pod \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " Jul 23 11:43:53.960486 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.960449 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2fhfj\" (UniqueName: \"kubernetes.io/projected/f1efdee5-e0b9-44c6-a001-566063caebca-kube-api-access-2fhfj\") pod \"f1efdee5-e0b9-44c6-a001-566063caebca\" (UID: \"f1efdee5-e0b9-44c6-a001-566063caebca\") " Jul 23 11:43:53.960531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.960518 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-extensions-socket-volume\") pod \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\" (UID: \"8ca4d0e5-ade9-4ab2-b573-ef833907e0e1\") " Jul 23 11:43:53.960789 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.960771 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dxl2\" (UniqueName: \"kubernetes.io/projected/a70456c7-2e83-4632-bd55-377ff4f0f1fa-kube-api-access-2dxl2\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:43:53.960953 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.960929 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" (UID: "8ca4d0e5-ade9-4ab2-b573-ef833907e0e1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:43:53.962640 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.962603 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-kube-api-access-c54x7" (OuterVolumeSpecName: "kube-api-access-c54x7") pod "8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" (UID: "8ca4d0e5-ade9-4ab2-b573-ef833907e0e1"). InnerVolumeSpecName "kube-api-access-c54x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:53.962693 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:53.962652 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1efdee5-e0b9-44c6-a001-566063caebca-kube-api-access-2fhfj" (OuterVolumeSpecName: "kube-api-access-2fhfj") pod "f1efdee5-e0b9-44c6-a001-566063caebca" (UID: "f1efdee5-e0b9-44c6-a001-566063caebca"). InnerVolumeSpecName "kube-api-access-2fhfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:43:53.963014 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:53.962989 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:54.061689 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.061605 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-extensions-socket-volume\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:43:54.061689 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.061688 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c54x7\" (UniqueName: \"kubernetes.io/projected/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1-kube-api-access-c54x7\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:43:54.061689 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.061704 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhfj\" (UniqueName: \"kubernetes.io/projected/f1efdee5-e0b9-44c6-a001-566063caebca-kube-api-access-2fhfj\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:43:54.115903 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.115868 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:54.125961 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.125933 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:43:54.270913 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.270879 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr"] Jul 23 11:43:54.274315 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:54.274290 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80cd0f30_ec4e_4524_bf78_844b956f98a2.slice/crio-4cf661558784ecf40371f168a2674a2d1dfb4f66c547a8abffcd0922c99b9b25 WatchSource:0}: Error finding container 4cf661558784ecf40371f168a2674a2d1dfb4f66c547a8abffcd0922c99b9b25: Status 404 returned error can't find the container with id 4cf661558784ecf40371f168a2674a2d1dfb4f66c547a8abffcd0922c99b9b25 Jul 23 11:43:54.297318 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.297210 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq"] Jul 23 11:43:54.300460 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:43:54.300432 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9490542d_4d00_4e2e_83a7_294aba9ab51b.slice/crio-3a6ab6dd201f7e1214d97ef67bb6d921a78dc10950397e9167979f19b0bd8a1f WatchSource:0}: Error finding container 3a6ab6dd201f7e1214d97ef67bb6d921a78dc10950397e9167979f19b0bd8a1f: Status 404 returned error can't find the container with id 3a6ab6dd201f7e1214d97ef67bb6d921a78dc10950397e9167979f19b0bd8a1f Jul 23 11:43:54.455405 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.455376 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" path="/var/lib/kubelet/pods/8ca4d0e5-ade9-4ab2-b573-ef833907e0e1/volumes" Jul 23 11:43:54.455770 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.455755 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" path="/var/lib/kubelet/pods/a70456c7-2e83-4632-bd55-377ff4f0f1fa/volumes" Jul 23 11:43:54.455962 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.455952 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1efdee5-e0b9-44c6-a001-566063caebca" path="/var/lib/kubelet/pods/f1efdee5-e0b9-44c6-a001-566063caebca/volumes" Jul 23 11:43:54.767240 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.767152 2554 generic.go:356] "Generic (PLEG): container finished" podID="f1efdee5-e0b9-44c6-a001-566063caebca" containerID="dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02" exitCode=0 Jul 23 11:43:54.767240 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.767206 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-pq2qz" Jul 23 11:43:54.767475 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.767260 2554 scope.go:117] "RemoveContainer" containerID="dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02" Jul 23 11:43:54.769293 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.769223 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" event={"ID":"9490542d-4d00-4e2e-83a7-294aba9ab51b","Type":"ContainerStarted","Data":"2600d139f88576b1da4dde813ecb4bcdc34689bbd8c7c0c472e03f3bb63ebfd1"} Jul 23 11:43:54.769486 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.769465 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:43:54.769649 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.769509 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" event={"ID":"9490542d-4d00-4e2e-83a7-294aba9ab51b","Type":"ContainerStarted","Data":"3a6ab6dd201f7e1214d97ef67bb6d921a78dc10950397e9167979f19b0bd8a1f"} Jul 23 11:43:54.770742 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.770723 2554 generic.go:356] "Generic (PLEG): container finished" podID="8ca4d0e5-ade9-4ab2-b573-ef833907e0e1" containerID="7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48" exitCode=0 Jul 23 11:43:54.770874 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.770776 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6d6v2" Jul 23 11:43:54.772540 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.772508 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:54.772654 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.772541 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" event={"ID":"80cd0f30-ec4e-4524-bf78-844b956f98a2","Type":"ContainerStarted","Data":"719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651"} Jul 23 11:43:54.772654 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.772575 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" event={"ID":"80cd0f30-ec4e-4524-bf78-844b956f98a2","Type":"ContainerStarted","Data":"4cf661558784ecf40371f168a2674a2d1dfb4f66c547a8abffcd0922c99b9b25"} Jul 23 11:43:54.772654 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.772651 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:43:54.777690 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.777675 2554 scope.go:117] "RemoveContainer" containerID="dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02" Jul 23 11:43:54.777952 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:54.777934 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02\": container with ID starting with dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02 not found: ID does not exist" containerID="dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02" Jul 23 11:43:54.778024 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.777964 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02"} err="failed to get container status \"dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02\": rpc error: code = NotFound desc = could not find container \"dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02\": container with ID starting with dd8928ee1107d683ded3bcc34d802dd3415cc3b2adf967dc47e65d1ebdbb4a02 not found: ID does not exist" Jul 23 11:43:54.778024 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.777992 2554 scope.go:117] "RemoveContainer" containerID="7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48" Jul 23 11:43:54.787512 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.787491 2554 scope.go:117] "RemoveContainer" containerID="7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48" Jul 23 11:43:54.787874 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:54.787851 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48\": container with ID starting with 7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48 not found: ID does not exist" containerID="7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48" Jul 23 11:43:54.787912 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.787884 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48"} err="failed to get container status \"7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48\": rpc error: code = NotFound desc = could not find container \"7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48\": container with ID starting with 7c5c3c24188c74930e48ce6e75c3a40755e9f48c0858a72c5a8a1f7070a80c48 not found: ID does not exist" Jul 23 11:43:54.792378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.792332 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" podStartSLOduration=1.7923150140000002 podStartE2EDuration="1.792315014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:54.790828305 +0000 UTC m=+482.896154735" watchObservedRunningTime="2026-07-23 11:43:54.792315014 +0000 UTC m=+482.897641403" Jul 23 11:43:54.793265 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:43:54.793233 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-2qs8g\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="a70456c7-2e83-4632-bd55-377ff4f0f1fa" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-2qs8g" Jul 23 11:43:54.822559 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:43:54.822492 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" podStartSLOduration=1.822472593 podStartE2EDuration="1.822472593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:43:54.82102666 +0000 UTC m=+482.926353070" watchObservedRunningTime="2026-07-23 11:43:54.822472593 +0000 UTC m=+482.927799005" Jul 23 11:44:05.787774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:05.787737 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:44:05.788274 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:05.787869 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-6t9fq" Jul 23 11:44:08.789526 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:08.789475 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7975bbc757-ldhlz" podUID="5e019307-72fd-4edf-bad3-1ea2e4c0e683" containerName="console" containerID="cri-o://5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762" gracePeriod=15 Jul 23 11:44:09.041273 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.041205 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:44:09.100084 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100043 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5499h\" (UniqueName: \"kubernetes.io/projected/5e019307-72fd-4edf-bad3-1ea2e4c0e683-kube-api-access-5499h\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100119 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-serving-cert\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100154 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-trusted-ca-bundle\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100176 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-config\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100207 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-service-ca\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100236 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-oauth-config\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100264 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-oauth-serving-cert\") pod \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\" (UID: \"5e019307-72fd-4edf-bad3-1ea2e4c0e683\") " Jul 23 11:44:09.100671 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100595 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-config" (OuterVolumeSpecName: "console-config") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:44:09.100774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100673 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-service-ca" (OuterVolumeSpecName: "service-ca") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:44:09.100774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100685 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:44:09.100938 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.100909 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 11:44:09.102533 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.102499 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:44:09.102905 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.102882 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e019307-72fd-4edf-bad3-1ea2e4c0e683-kube-api-access-5499h" (OuterVolumeSpecName: "kube-api-access-5499h") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "kube-api-access-5499h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:44:09.102978 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.102908 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5e019307-72fd-4edf-bad3-1ea2e4c0e683" (UID: "5e019307-72fd-4edf-bad3-1ea2e4c0e683"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:44:09.201345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201300 2554 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-trusted-ca-bundle\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.201345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201335 2554 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.201345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201344 2554 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-service-ca\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.201345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201351 2554 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-oauth-config\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.201613 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201363 2554 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5e019307-72fd-4edf-bad3-1ea2e4c0e683-oauth-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.201613 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201372 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5499h\" (UniqueName: \"kubernetes.io/projected/5e019307-72fd-4edf-bad3-1ea2e4c0e683-kube-api-access-5499h\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.201613 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.201381 2554 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5e019307-72fd-4edf-bad3-1ea2e4c0e683-console-serving-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:09.856200 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.856163 2554 generic.go:356] "Generic (PLEG): container finished" podID="5e019307-72fd-4edf-bad3-1ea2e4c0e683" containerID="5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762" exitCode=0 Jul 23 11:44:09.856568 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.856216 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7975bbc757-ldhlz" Jul 23 11:44:09.856568 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.856247 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975bbc757-ldhlz" event={"ID":"5e019307-72fd-4edf-bad3-1ea2e4c0e683","Type":"ContainerDied","Data":"5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762"} Jul 23 11:44:09.856568 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.856309 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7975bbc757-ldhlz" event={"ID":"5e019307-72fd-4edf-bad3-1ea2e4c0e683","Type":"ContainerDied","Data":"cd87e45043d9973848c45c28feb05c3fc76fc6ff7dde248ef05f3722be7efd68"} Jul 23 11:44:09.856568 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.856330 2554 scope.go:117] "RemoveContainer" containerID="5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762" Jul 23 11:44:09.867550 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.867528 2554 scope.go:117] "RemoveContainer" containerID="5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762" Jul 23 11:44:09.868283 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:44:09.868252 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762\": container with ID starting with 5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762 not found: ID does not exist" containerID="5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762" Jul 23 11:44:09.868386 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.868295 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762"} err="failed to get container status \"5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762\": rpc error: code = NotFound desc = could not find container \"5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762\": container with ID starting with 5d9cc40707705d2bd4fcb6cde1703f4524b5430a454e1f8cbb759ed65f179762 not found: ID does not exist" Jul 23 11:44:09.879988 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.879961 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7975bbc757-ldhlz"] Jul 23 11:44:09.885748 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:09.885720 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7975bbc757-ldhlz"] Jul 23 11:44:10.453454 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:10.453417 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e019307-72fd-4edf-bad3-1ea2e4c0e683" path="/var/lib/kubelet/pods/5e019307-72fd-4edf-bad3-1ea2e4c0e683/volumes" Jul 23 11:44:22.413297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.413258 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr"] Jul 23 11:44:22.413751 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.413476 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" podUID="80cd0f30-ec4e-4524-bf78-844b956f98a2" containerName="manager" containerID="cri-o://719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651" gracePeriod=10 Jul 23 11:44:22.666879 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.666784 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v"] Jul 23 11:44:22.667229 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.667214 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e019307-72fd-4edf-bad3-1ea2e4c0e683" containerName="console" Jul 23 11:44:22.667229 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.667230 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e019307-72fd-4edf-bad3-1ea2e4c0e683" containerName="console" Jul 23 11:44:22.667335 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.667305 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e019307-72fd-4edf-bad3-1ea2e4c0e683" containerName="console" Jul 23 11:44:22.670268 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.670248 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.670993 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.670966 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:44:22.683729 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.683701 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v"] Jul 23 11:44:22.828160 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.828123 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd0f30-ec4e-4524-bf78-844b956f98a2-extensions-socket-volume\") pod \"80cd0f30-ec4e-4524-bf78-844b956f98a2\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " Jul 23 11:44:22.828347 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.828202 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx9tf\" (UniqueName: \"kubernetes.io/projected/80cd0f30-ec4e-4524-bf78-844b956f98a2-kube-api-access-fx9tf\") pod \"80cd0f30-ec4e-4524-bf78-844b956f98a2\" (UID: \"80cd0f30-ec4e-4524-bf78-844b956f98a2\") " Jul 23 11:44:22.828411 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.828391 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e744f37-fa15-45f3-9b6c-293d7570f93f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lgp6v\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.828453 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.828436 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77plp\" (UniqueName: \"kubernetes.io/projected/5e744f37-fa15-45f3-9b6c-293d7570f93f-kube-api-access-77plp\") pod \"kuadrant-operator-controller-manager-86878c769f-lgp6v\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.828541 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.828518 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80cd0f30-ec4e-4524-bf78-844b956f98a2-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "80cd0f30-ec4e-4524-bf78-844b956f98a2" (UID: "80cd0f30-ec4e-4524-bf78-844b956f98a2"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:44:22.830531 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.830503 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80cd0f30-ec4e-4524-bf78-844b956f98a2-kube-api-access-fx9tf" (OuterVolumeSpecName: "kube-api-access-fx9tf") pod "80cd0f30-ec4e-4524-bf78-844b956f98a2" (UID: "80cd0f30-ec4e-4524-bf78-844b956f98a2"). InnerVolumeSpecName "kube-api-access-fx9tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:44:22.923045 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.922935 2554 generic.go:356] "Generic (PLEG): container finished" podID="80cd0f30-ec4e-4524-bf78-844b956f98a2" containerID="719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651" exitCode=0 Jul 23 11:44:22.923045 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.922996 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" Jul 23 11:44:22.923045 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.923029 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" event={"ID":"80cd0f30-ec4e-4524-bf78-844b956f98a2","Type":"ContainerDied","Data":"719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651"} Jul 23 11:44:22.923338 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.923097 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr" event={"ID":"80cd0f30-ec4e-4524-bf78-844b956f98a2","Type":"ContainerDied","Data":"4cf661558784ecf40371f168a2674a2d1dfb4f66c547a8abffcd0922c99b9b25"} Jul 23 11:44:22.923338 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.923129 2554 scope.go:117] "RemoveContainer" containerID="719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651" Jul 23 11:44:22.929009 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.928979 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e744f37-fa15-45f3-9b6c-293d7570f93f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lgp6v\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.929154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.929035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77plp\" (UniqueName: \"kubernetes.io/projected/5e744f37-fa15-45f3-9b6c-293d7570f93f-kube-api-access-77plp\") pod \"kuadrant-operator-controller-manager-86878c769f-lgp6v\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.929154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.929095 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/80cd0f30-ec4e-4524-bf78-844b956f98a2-extensions-socket-volume\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:22.929154 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.929112 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx9tf\" (UniqueName: \"kubernetes.io/projected/80cd0f30-ec4e-4524-bf78-844b956f98a2-kube-api-access-fx9tf\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:22.929436 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.929412 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e744f37-fa15-45f3-9b6c-293d7570f93f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-lgp6v\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.933664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.933612 2554 scope.go:117] "RemoveContainer" containerID="719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651" Jul 23 11:44:22.934009 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:44:22.933986 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651\": container with ID starting with 719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651 not found: ID does not exist" containerID="719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651" Jul 23 11:44:22.934086 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.934019 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651"} err="failed to get container status \"719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651\": rpc error: code = NotFound desc = could not find container \"719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651\": container with ID starting with 719d630102e99fa0a24d5fbbf101113bceb7ba7746a9d4f3f4c52dccb6acf651 not found: ID does not exist" Jul 23 11:44:22.940275 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.940245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77plp\" (UniqueName: \"kubernetes.io/projected/5e744f37-fa15-45f3-9b6c-293d7570f93f-kube-api-access-77plp\") pod \"kuadrant-operator-controller-manager-86878c769f-lgp6v\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:22.946013 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.945984 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr"] Jul 23 11:44:22.950189 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.950160 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-skvzr"] Jul 23 11:44:22.980345 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:22.980310 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:23.131179 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:23.131146 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v"] Jul 23 11:44:23.134883 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:44:23.134837 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e744f37_fa15_45f3_9b6c_293d7570f93f.slice/crio-35dcc6df040d8dd86466bbc6a45b07d9514e24f2653653128f5a64632005c9c7 WatchSource:0}: Error finding container 35dcc6df040d8dd86466bbc6a45b07d9514e24f2653653128f5a64632005c9c7: Status 404 returned error can't find the container with id 35dcc6df040d8dd86466bbc6a45b07d9514e24f2653653128f5a64632005c9c7 Jul 23 11:44:23.930381 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:23.930342 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" event={"ID":"5e744f37-fa15-45f3-9b6c-293d7570f93f","Type":"ContainerStarted","Data":"7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9"} Jul 23 11:44:23.930898 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:23.930409 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" event={"ID":"5e744f37-fa15-45f3-9b6c-293d7570f93f","Type":"ContainerStarted","Data":"35dcc6df040d8dd86466bbc6a45b07d9514e24f2653653128f5a64632005c9c7"} Jul 23 11:44:23.930898 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:23.930559 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:23.971693 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:23.971613 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" podStartSLOduration=1.971593587 podStartE2EDuration="1.971593587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:44:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:23.969024257 +0000 UTC m=+512.074350671" watchObservedRunningTime="2026-07-23 11:44:23.971593587 +0000 UTC m=+512.076920053" Jul 23 11:44:24.459037 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:24.459002 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80cd0f30-ec4e-4524-bf78-844b956f98a2" path="/var/lib/kubelet/pods/80cd0f30-ec4e-4524-bf78-844b956f98a2/volumes" Jul 23 11:44:34.939903 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:34.939868 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:44:38.815920 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.815876 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp"] Jul 23 11:44:38.816388 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.816315 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80cd0f30-ec4e-4524-bf78-844b956f98a2" containerName="manager" Jul 23 11:44:38.816388 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.816328 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="80cd0f30-ec4e-4524-bf78-844b956f98a2" containerName="manager" Jul 23 11:44:38.816460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.816410 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="80cd0f30-ec4e-4524-bf78-844b956f98a2" containerName="manager" Jul 23 11:44:38.820923 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.820900 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.823212 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.823188 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-2phkg\"" Jul 23 11:44:38.830873 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.830834 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp"] Jul 23 11:44:38.873793 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873751 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.873971 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873831 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.873971 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873860 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.873971 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873882 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.873971 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873954 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5ngm\" (UniqueName: \"kubernetes.io/projected/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-kube-api-access-f5ngm\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.874103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873979 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.874103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.873996 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.874103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.874051 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.874103 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.874088 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.974984 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.974949 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975176 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.974991 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975176 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975120 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975289 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975211 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5ngm\" (UniqueName: \"kubernetes.io/projected/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-kube-api-access-f5ngm\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975289 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975240 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975289 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975257 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975289 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975284 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975457 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975457 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975360 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975457 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975383 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-data\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975557 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-workload-certs\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975656 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975648 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-credential-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.975930 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975902 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-workload-socket\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.976010 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.975909 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.977715 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.977690 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-envoy\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.978052 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.978033 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.985030 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.985008 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5ngm\" (UniqueName: \"kubernetes.io/projected/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-kube-api-access-f5ngm\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:38.994434 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:38.994408 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d1ea3e1-59ee-4534-8b27-8a054245e1d1-istio-token\") pod \"maas-default-gateway-openshift-default-56654dcf7-bxzlp\" (UID: \"2d1ea3e1-59ee-4534-8b27-8a054245e1d1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:39.133264 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:39.133162 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:39.288927 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:39.288901 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp"] Jul 23 11:44:39.291100 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:44:39.291069 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1ea3e1_59ee_4534_8b27_8a054245e1d1.slice/crio-ad4af472c1b3a00b8365928e2d6ecd2099f5b8959c19bb5f874999e2ebd2b563 WatchSource:0}: Error finding container ad4af472c1b3a00b8365928e2d6ecd2099f5b8959c19bb5f874999e2ebd2b563: Status 404 returned error can't find the container with id ad4af472c1b3a00b8365928e2d6ecd2099f5b8959c19bb5f874999e2ebd2b563 Jul 23 11:44:39.293163 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:39.293132 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:44:39.293260 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:39.293219 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:44:39.293320 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:39.293261 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:44:40.014778 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:40.014739 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" event={"ID":"2d1ea3e1-59ee-4534-8b27-8a054245e1d1","Type":"ContainerStarted","Data":"634234eed676ef3a9ea8faa0494f6481999940b19f1f86f8d23e401b2cdee172"} Jul 23 11:44:40.015209 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:40.014799 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" event={"ID":"2d1ea3e1-59ee-4534-8b27-8a054245e1d1","Type":"ContainerStarted","Data":"ad4af472c1b3a00b8365928e2d6ecd2099f5b8959c19bb5f874999e2ebd2b563"} Jul 23 11:44:40.040314 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:40.040234 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" podStartSLOduration=2.040207956 podStartE2EDuration="2.040207956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:44:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:40.036637123 +0000 UTC m=+528.141963525" watchObservedRunningTime="2026-07-23 11:44:40.040207956 +0000 UTC m=+528.145534369" Jul 23 11:44:40.133979 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:40.133934 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:40.139021 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:40.138991 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:41.020060 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:41.020028 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:41.021198 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:41.021176 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:44:43.169661 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.169567 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-ztkm8"] Jul 23 11:44:43.173551 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.173525 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:43.175881 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.175856 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-248mc\"" Jul 23 11:44:43.181017 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.180979 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ztkm8"] Jul 23 11:44:43.215185 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.215148 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f89h9\" (UniqueName: \"kubernetes.io/projected/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9-kube-api-access-f89h9\") pod \"authorino-58cd955675-ztkm8\" (UID: \"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9\") " pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:43.316268 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.316233 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f89h9\" (UniqueName: \"kubernetes.io/projected/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9-kube-api-access-f89h9\") pod \"authorino-58cd955675-ztkm8\" (UID: \"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9\") " pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:43.324871 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.324842 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f89h9\" (UniqueName: \"kubernetes.io/projected/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9-kube-api-access-f89h9\") pod \"authorino-58cd955675-ztkm8\" (UID: \"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9\") " pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:43.485536 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.485447 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:43.657690 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:43.657639 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-ztkm8"] Jul 23 11:44:43.660979 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:44:43.660946 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda25f6cc5_c53a_4e2b_aac5_f9a06810e2b9.slice/crio-ba59b17f7cc078bba109f9c311964ce9911866856af64de062182494f7ec20e7 WatchSource:0}: Error finding container ba59b17f7cc078bba109f9c311964ce9911866856af64de062182494f7ec20e7: Status 404 returned error can't find the container with id ba59b17f7cc078bba109f9c311964ce9911866856af64de062182494f7ec20e7 Jul 23 11:44:44.037134 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:44.037095 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ztkm8" event={"ID":"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9","Type":"ContainerStarted","Data":"ba59b17f7cc078bba109f9c311964ce9911866856af64de062182494f7ec20e7"} Jul 23 11:44:47.057369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:47.057328 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ztkm8" event={"ID":"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9","Type":"ContainerStarted","Data":"8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6"} Jul 23 11:44:47.075352 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:47.075284 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-ztkm8" podStartSLOduration=0.875164106 podStartE2EDuration="4.07526626s" totalImagesPullingTime="3.200102154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:44:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:44:47.073256578 +0000 UTC m=+535.178582987" watchObservedRunningTime="2026-07-23 11:44:47.07526626 +0000 UTC m=+535.180592672" Jul 23 11:44:48.581439 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:48.581402 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ztkm8"] Jul 23 11:44:49.069488 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:49.069441 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-ztkm8" podUID="a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" containerName="authorino" containerID="cri-o://8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6" gracePeriod=30 Jul 23 11:44:49.327205 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:49.327146 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:49.373359 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:49.373325 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f89h9\" (UniqueName: \"kubernetes.io/projected/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9-kube-api-access-f89h9\") pod \"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9\" (UID: \"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9\") " Jul 23 11:44:49.375606 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:49.375570 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9-kube-api-access-f89h9" (OuterVolumeSpecName: "kube-api-access-f89h9") pod "a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" (UID: "a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9"). InnerVolumeSpecName "kube-api-access-f89h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:44:49.474148 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:49.474108 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f89h9\" (UniqueName: \"kubernetes.io/projected/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9-kube-api-access-f89h9\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:44:50.075681 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.075640 2554 generic.go:356] "Generic (PLEG): container finished" podID="a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" containerID="8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6" exitCode=0 Jul 23 11:44:50.075681 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.075653 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ztkm8" event={"ID":"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9","Type":"ContainerDied","Data":"8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6"} Jul 23 11:44:50.076191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.075694 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-ztkm8" Jul 23 11:44:50.076191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.075724 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-ztkm8" event={"ID":"a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9","Type":"ContainerDied","Data":"ba59b17f7cc078bba109f9c311964ce9911866856af64de062182494f7ec20e7"} Jul 23 11:44:50.076191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.075750 2554 scope.go:117] "RemoveContainer" containerID="8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6" Jul 23 11:44:50.084992 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.084975 2554 scope.go:117] "RemoveContainer" containerID="8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6" Jul 23 11:44:50.085245 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:44:50.085229 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6\": container with ID starting with 8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6 not found: ID does not exist" containerID="8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6" Jul 23 11:44:50.085311 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.085258 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6"} err="failed to get container status \"8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6\": rpc error: code = NotFound desc = could not find container \"8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6\": container with ID starting with 8646440c16e7a2c3acfe8bb8ea1cf468cc8249d16c55b88bd3403176ebe47bc6 not found: ID does not exist" Jul 23 11:44:50.097461 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.097432 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-ztkm8"] Jul 23 11:44:50.101285 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.101260 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-ztkm8"] Jul 23 11:44:50.454028 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:44:50.453925 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" path="/var/lib/kubelet/pods/a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9/volumes" Jul 23 11:45:03.000681 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.000638 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc"] Jul 23 11:45:03.001118 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.001092 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" containerName="authorino" Jul 23 11:45:03.001118 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.001106 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" containerName="authorino" Jul 23 11:45:03.001230 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.001183 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="a25f6cc5-c53a-4e2b-aac5-f9a06810e2b9" containerName="authorino" Jul 23 11:45:03.005801 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.005778 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.010259 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.010234 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-wfdrw\"" Jul 23 11:45:03.011029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.011000 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Jul 23 11:45:03.011029 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.011023 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Jul 23 11:45:03.011231 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.011044 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 23 11:45:03.027846 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.027811 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc"] Jul 23 11:45:03.102472 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.102436 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59hd6\" (UniqueName: \"kubernetes.io/projected/36651474-2053-4546-a022-c97873821378-kube-api-access-59hd6\") pod \"postgres-69fb4594cf-gxdcc\" (UID: \"36651474-2053-4546-a022-c97873821378\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.102696 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.102482 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36651474-2053-4546-a022-c97873821378-data\") pod \"postgres-69fb4594cf-gxdcc\" (UID: \"36651474-2053-4546-a022-c97873821378\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.203560 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.203520 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59hd6\" (UniqueName: \"kubernetes.io/projected/36651474-2053-4546-a022-c97873821378-kube-api-access-59hd6\") pod \"postgres-69fb4594cf-gxdcc\" (UID: \"36651474-2053-4546-a022-c97873821378\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.203560 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.203572 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36651474-2053-4546-a022-c97873821378-data\") pod \"postgres-69fb4594cf-gxdcc\" (UID: \"36651474-2053-4546-a022-c97873821378\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.204087 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.204066 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/36651474-2053-4546-a022-c97873821378-data\") pod \"postgres-69fb4594cf-gxdcc\" (UID: \"36651474-2053-4546-a022-c97873821378\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.214734 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.214707 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-59hd6\" (UniqueName: \"kubernetes.io/projected/36651474-2053-4546-a022-c97873821378-kube-api-access-59hd6\") pod \"postgres-69fb4594cf-gxdcc\" (UID: \"36651474-2053-4546-a022-c97873821378\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.316374 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.316341 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:03.474848 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:03.474813 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc"] Jul 23 11:45:03.477339 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:45:03.477310 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36651474_2053_4546_a022_c97873821378.slice/crio-3ed3daea9022267e2f0bff0c9797145ff67f9122bd5e22c1acc4396e9f6c8405 WatchSource:0}: Error finding container 3ed3daea9022267e2f0bff0c9797145ff67f9122bd5e22c1acc4396e9f6c8405: Status 404 returned error can't find the container with id 3ed3daea9022267e2f0bff0c9797145ff67f9122bd5e22c1acc4396e9f6c8405 Jul 23 11:45:04.148253 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:04.148207 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" event={"ID":"36651474-2053-4546-a022-c97873821378","Type":"ContainerStarted","Data":"3ed3daea9022267e2f0bff0c9797145ff67f9122bd5e22c1acc4396e9f6c8405"} Jul 23 11:45:09.178896 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:09.178824 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" event={"ID":"36651474-2053-4546-a022-c97873821378","Type":"ContainerStarted","Data":"bc513fcf178e14e7ed5abc59aa3e1287bb6e853fee666cde0d9b882150e010fe"} Jul 23 11:45:09.179463 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:09.178957 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:15.217091 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.217052 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" Jul 23 11:45:15.267736 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.267672 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gxdcc" podStartSLOduration=8.343998702 podStartE2EDuration="13.267653636s" totalImagesPullingTime="4.923654934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:09.200072249 +0000 UTC m=+557.305398661" watchObservedRunningTime="2026-07-23 11:45:15.267653636 +0000 UTC m=+563.372980049" Jul 23 11:45:15.853824 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.853783 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mt4hb"] Jul 23 11:45:15.857597 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.857567 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:15.860290 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.860264 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-248mc\"" Jul 23 11:45:15.872176 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.872136 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mt4hb"] Jul 23 11:45:15.927072 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:15.927038 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6md5c\" (UniqueName: \"kubernetes.io/projected/7efa7320-901f-4b1c-80f5-22f90a29953a-kube-api-access-6md5c\") pod \"authorino-5d4b67c446-mt4hb\" (UID: \"7efa7320-901f-4b1c-80f5-22f90a29953a\") " pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:16.028109 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.028061 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6md5c\" (UniqueName: \"kubernetes.io/projected/7efa7320-901f-4b1c-80f5-22f90a29953a-kube-api-access-6md5c\") pod \"authorino-5d4b67c446-mt4hb\" (UID: \"7efa7320-901f-4b1c-80f5-22f90a29953a\") " pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:16.037412 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.037379 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6md5c\" (UniqueName: \"kubernetes.io/projected/7efa7320-901f-4b1c-80f5-22f90a29953a-kube-api-access-6md5c\") pod \"authorino-5d4b67c446-mt4hb\" (UID: \"7efa7320-901f-4b1c-80f5-22f90a29953a\") " pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:16.099638 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.099590 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mt4hb"] Jul 23 11:45:16.099935 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.099923 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:16.125914 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.125875 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7cf5465945-thmkz"] Jul 23 11:45:16.129827 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.129796 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:16.142797 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.142758 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7cf5465945-thmkz"] Jul 23 11:45:16.229838 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.229804 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxncz\" (UniqueName: \"kubernetes.io/projected/d304a614-c960-403e-8a87-9260701bd98a-kube-api-access-zxncz\") pod \"authorino-7cf5465945-thmkz\" (UID: \"d304a614-c960-403e-8a87-9260701bd98a\") " pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:16.258213 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.258178 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mt4hb"] Jul 23 11:45:16.262169 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:45:16.262134 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7efa7320_901f_4b1c_80f5_22f90a29953a.slice/crio-1e331f299dfc2651ad4be3f2b95a5fbf2cb50247406964a461cb6047f460516d WatchSource:0}: Error finding container 1e331f299dfc2651ad4be3f2b95a5fbf2cb50247406964a461cb6047f460516d: Status 404 returned error can't find the container with id 1e331f299dfc2651ad4be3f2b95a5fbf2cb50247406964a461cb6047f460516d Jul 23 11:45:16.331080 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.331036 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxncz\" (UniqueName: \"kubernetes.io/projected/d304a614-c960-403e-8a87-9260701bd98a-kube-api-access-zxncz\") pod \"authorino-7cf5465945-thmkz\" (UID: \"d304a614-c960-403e-8a87-9260701bd98a\") " pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:16.341644 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.341601 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxncz\" (UniqueName: \"kubernetes.io/projected/d304a614-c960-403e-8a87-9260701bd98a-kube-api-access-zxncz\") pod \"authorino-7cf5465945-thmkz\" (UID: \"d304a614-c960-403e-8a87-9260701bd98a\") " pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:16.395593 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.395497 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cf5465945-thmkz"] Jul 23 11:45:16.395843 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.395829 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:16.428250 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.428212 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5679b76fd9-rmznc"] Jul 23 11:45:16.431911 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.431887 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.434925 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.434897 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 23 11:45:16.443012 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.442940 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5679b76fd9-rmznc"] Jul 23 11:45:16.533282 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.533221 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-tls-cert\") pod \"authorino-5679b76fd9-rmznc\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.533282 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.533262 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdwvk\" (UniqueName: \"kubernetes.io/projected/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-kube-api-access-vdwvk\") pod \"authorino-5679b76fd9-rmznc\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.554262 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.554235 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cf5465945-thmkz"] Jul 23 11:45:16.556299 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:45:16.556270 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd304a614_c960_403e_8a87_9260701bd98a.slice/crio-817dbd59518e5ee720a12c58bbf9ccd4fa90595120c90791aa5b7434c37fc9c9 WatchSource:0}: Error finding container 817dbd59518e5ee720a12c58bbf9ccd4fa90595120c90791aa5b7434c37fc9c9: Status 404 returned error can't find the container with id 817dbd59518e5ee720a12c58bbf9ccd4fa90595120c90791aa5b7434c37fc9c9 Jul 23 11:45:16.633748 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.633715 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-tls-cert\") pod \"authorino-5679b76fd9-rmznc\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.633918 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.633759 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdwvk\" (UniqueName: \"kubernetes.io/projected/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-kube-api-access-vdwvk\") pod \"authorino-5679b76fd9-rmznc\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.637310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.637285 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-tls-cert\") pod \"authorino-5679b76fd9-rmznc\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.646822 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.646746 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdwvk\" (UniqueName: \"kubernetes.io/projected/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-kube-api-access-vdwvk\") pod \"authorino-5679b76fd9-rmznc\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.744840 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.744787 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:45:16.901557 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:16.901460 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5679b76fd9-rmznc"] Jul 23 11:45:16.903918 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:45:16.903887 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod029bd78c_7e0f_4e30_a983_7b80da6e6bb5.slice/crio-1b1aa1882b857ddae8e047451fc3b43207f6975bd8130e8d2b701c97351647d8 WatchSource:0}: Error finding container 1b1aa1882b857ddae8e047451fc3b43207f6975bd8130e8d2b701c97351647d8: Status 404 returned error can't find the container with id 1b1aa1882b857ddae8e047451fc3b43207f6975bd8130e8d2b701c97351647d8 Jul 23 11:45:17.224601 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.224493 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5679b76fd9-rmznc" event={"ID":"029bd78c-7e0f-4e30-a983-7b80da6e6bb5","Type":"ContainerStarted","Data":"1b1aa1882b857ddae8e047451fc3b43207f6975bd8130e8d2b701c97351647d8"} Jul 23 11:45:17.226093 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.226064 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf5465945-thmkz" event={"ID":"d304a614-c960-403e-8a87-9260701bd98a","Type":"ContainerStarted","Data":"4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366"} Jul 23 11:45:17.226266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.226102 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf5465945-thmkz" event={"ID":"d304a614-c960-403e-8a87-9260701bd98a","Type":"ContainerStarted","Data":"817dbd59518e5ee720a12c58bbf9ccd4fa90595120c90791aa5b7434c37fc9c9"} Jul 23 11:45:17.226266 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.226126 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7cf5465945-thmkz" podUID="d304a614-c960-403e-8a87-9260701bd98a" containerName="authorino" containerID="cri-o://4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366" gracePeriod=30 Jul 23 11:45:17.227680 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.227652 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" event={"ID":"7efa7320-901f-4b1c-80f5-22f90a29953a","Type":"ContainerStarted","Data":"13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f"} Jul 23 11:45:17.227787 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.227690 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" event={"ID":"7efa7320-901f-4b1c-80f5-22f90a29953a","Type":"ContainerStarted","Data":"1e331f299dfc2651ad4be3f2b95a5fbf2cb50247406964a461cb6047f460516d"} Jul 23 11:45:17.227787 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.227761 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" podUID="7efa7320-901f-4b1c-80f5-22f90a29953a" containerName="authorino" containerID="cri-o://13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f" gracePeriod=30 Jul 23 11:45:17.245213 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.245157 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7cf5465945-thmkz" podStartSLOduration=0.780057517 podStartE2EDuration="1.245136183s" totalImagesPullingTime="465.078666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:17.243214594 +0000 UTC m=+565.348541009" watchObservedRunningTime="2026-07-23 11:45:17.245136183 +0000 UTC m=+565.350462593" Jul 23 11:45:17.264292 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.264222 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" podStartSLOduration=1.867229548 podStartE2EDuration="2.26420436s" totalImagesPullingTime="396.974812ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:17.261707239 +0000 UTC m=+565.367033654" watchObservedRunningTime="2026-07-23 11:45:17.26420436 +0000 UTC m=+565.369530770" Jul 23 11:45:17.509987 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.509957 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:17.514247 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.514227 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:17.540416 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.540382 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6md5c\" (UniqueName: \"kubernetes.io/projected/7efa7320-901f-4b1c-80f5-22f90a29953a-kube-api-access-6md5c\") pod \"7efa7320-901f-4b1c-80f5-22f90a29953a\" (UID: \"7efa7320-901f-4b1c-80f5-22f90a29953a\") " Jul 23 11:45:17.540661 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.540427 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxncz\" (UniqueName: \"kubernetes.io/projected/d304a614-c960-403e-8a87-9260701bd98a-kube-api-access-zxncz\") pod \"d304a614-c960-403e-8a87-9260701bd98a\" (UID: \"d304a614-c960-403e-8a87-9260701bd98a\") " Jul 23 11:45:17.543086 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.543052 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7efa7320-901f-4b1c-80f5-22f90a29953a-kube-api-access-6md5c" (OuterVolumeSpecName: "kube-api-access-6md5c") pod "7efa7320-901f-4b1c-80f5-22f90a29953a" (UID: "7efa7320-901f-4b1c-80f5-22f90a29953a"). InnerVolumeSpecName "kube-api-access-6md5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:45:17.543086 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.543062 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d304a614-c960-403e-8a87-9260701bd98a-kube-api-access-zxncz" (OuterVolumeSpecName: "kube-api-access-zxncz") pod "d304a614-c960-403e-8a87-9260701bd98a" (UID: "d304a614-c960-403e-8a87-9260701bd98a"). InnerVolumeSpecName "kube-api-access-zxncz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:45:17.641105 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.641069 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6md5c\" (UniqueName: \"kubernetes.io/projected/7efa7320-901f-4b1c-80f5-22f90a29953a-kube-api-access-6md5c\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:45:17.641105 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:17.641102 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxncz\" (UniqueName: \"kubernetes.io/projected/d304a614-c960-403e-8a87-9260701bd98a-kube-api-access-zxncz\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:45:18.234325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.234284 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5679b76fd9-rmznc" event={"ID":"029bd78c-7e0f-4e30-a983-7b80da6e6bb5","Type":"ContainerStarted","Data":"3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70"} Jul 23 11:45:18.235609 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.235581 2554 generic.go:356] "Generic (PLEG): container finished" podID="d304a614-c960-403e-8a87-9260701bd98a" containerID="4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366" exitCode=0 Jul 23 11:45:18.235774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.235653 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7cf5465945-thmkz" Jul 23 11:45:18.235774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.235656 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf5465945-thmkz" event={"ID":"d304a614-c960-403e-8a87-9260701bd98a","Type":"ContainerDied","Data":"4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366"} Jul 23 11:45:18.235774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.235694 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7cf5465945-thmkz" event={"ID":"d304a614-c960-403e-8a87-9260701bd98a","Type":"ContainerDied","Data":"817dbd59518e5ee720a12c58bbf9ccd4fa90595120c90791aa5b7434c37fc9c9"} Jul 23 11:45:18.235774 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.235717 2554 scope.go:117] "RemoveContainer" containerID="4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366" Jul 23 11:45:18.236996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.236894 2554 generic.go:356] "Generic (PLEG): container finished" podID="7efa7320-901f-4b1c-80f5-22f90a29953a" containerID="13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f" exitCode=0 Jul 23 11:45:18.236996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.236938 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" event={"ID":"7efa7320-901f-4b1c-80f5-22f90a29953a","Type":"ContainerDied","Data":"13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f"} Jul 23 11:45:18.236996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.236953 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" Jul 23 11:45:18.236996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.236981 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mt4hb" event={"ID":"7efa7320-901f-4b1c-80f5-22f90a29953a","Type":"ContainerDied","Data":"1e331f299dfc2651ad4be3f2b95a5fbf2cb50247406964a461cb6047f460516d"} Jul 23 11:45:18.246856 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.246758 2554 scope.go:117] "RemoveContainer" containerID="4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366" Jul 23 11:45:18.247248 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:45:18.247227 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366\": container with ID starting with 4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366 not found: ID does not exist" containerID="4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366" Jul 23 11:45:18.247325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.247257 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366"} err="failed to get container status \"4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366\": rpc error: code = NotFound desc = could not find container \"4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366\": container with ID starting with 4463f20144a079de7831b4c37ba7b79e1a426f52f7256e2f775ddd5e1c701366 not found: ID does not exist" Jul 23 11:45:18.247325 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.247282 2554 scope.go:117] "RemoveContainer" containerID="13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f" Jul 23 11:45:18.253268 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.253194 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5679b76fd9-rmznc" podStartSLOduration=1.6549774830000001 podStartE2EDuration="2.253170317s" totalImagesPullingTime="598.192834ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:45:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:45:18.252182446 +0000 UTC m=+566.357508858" watchObservedRunningTime="2026-07-23 11:45:18.253170317 +0000 UTC m=+566.358496729" Jul 23 11:45:18.258200 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.258180 2554 scope.go:117] "RemoveContainer" containerID="13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f" Jul 23 11:45:18.258529 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:45:18.258504 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f\": container with ID starting with 13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f not found: ID does not exist" containerID="13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f" Jul 23 11:45:18.258569 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.258539 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f"} err="failed to get container status \"13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f\": rpc error: code = NotFound desc = could not find container \"13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f\": container with ID starting with 13a1bc026c0b24ce826910320876160716bce67ffbe6acdbd52da9796488c19f not found: ID does not exist" Jul 23 11:45:18.269539 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.269510 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7cf5465945-thmkz"] Jul 23 11:45:18.273428 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.273399 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7cf5465945-thmkz"] Jul 23 11:45:18.290980 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.290938 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mt4hb"] Jul 23 11:45:18.296209 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.296127 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mt4hb"] Jul 23 11:45:18.454207 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.454168 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7efa7320-901f-4b1c-80f5-22f90a29953a" path="/var/lib/kubelet/pods/7efa7320-901f-4b1c-80f5-22f90a29953a/volumes" Jul 23 11:45:18.454513 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:45:18.454499 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d304a614-c960-403e-8a87-9260701bd98a" path="/var/lib/kubelet/pods/d304a614-c960-403e-8a87-9260701bd98a/volumes" Jul 23 11:46:05.496728 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:05.496694 2554 generic.go:356] "Generic (PLEG): container finished" podID="2d1ea3e1-59ee-4534-8b27-8a054245e1d1" containerID="634234eed676ef3a9ea8faa0494f6481999940b19f1f86f8d23e401b2cdee172" exitCode=0 Jul 23 11:46:05.497170 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:05.496773 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" event={"ID":"2d1ea3e1-59ee-4534-8b27-8a054245e1d1","Type":"ContainerDied","Data":"634234eed676ef3a9ea8faa0494f6481999940b19f1f86f8d23e401b2cdee172"} Jul 23 11:46:05.497170 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:05.497159 2554 scope.go:117] "RemoveContainer" containerID="634234eed676ef3a9ea8faa0494f6481999940b19f1f86f8d23e401b2cdee172" Jul 23 11:46:05.498211 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:05.498184 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:46:05.498277 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:05.498258 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:46:05.498316 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:05.498293 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 11:46:06.507231 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:06.507189 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" event={"ID":"2d1ea3e1-59ee-4534-8b27-8a054245e1d1","Type":"ContainerStarted","Data":"ce7a074f6601ff1741925433d73f97c73539fc5e26ce3aa8350b72533ad1dbf9"} Jul 23 11:46:07.133704 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:07.133665 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:46:07.135011 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:07.134984 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 11:46:07.135124 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:07.135036 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" podUID="2d1ea3e1-59ee-4534-8b27-8a054245e1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 11:46:08.134201 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:08.134157 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 11:46:08.134605 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:08.134219 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" podUID="2d1ea3e1-59ee-4534-8b27-8a054245e1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 11:46:09.134595 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:09.134557 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 11:46:09.135005 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:09.134649 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" podUID="2d1ea3e1-59ee-4534-8b27-8a054245e1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 11:46:10.134206 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:10.134164 2554 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-56654dcf7-bxzlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" start-of-body= Jul 23 11:46:10.134426 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:10.134242 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" podUID="2d1ea3e1-59ee-4534-8b27-8a054245e1d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.43:15021/healthz/ready\": dial tcp 10.134.0.43:15021: connect: connection refused" Jul 23 11:46:11.020822 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:11.020791 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:46:11.138095 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:11.138062 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:46:11.535005 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:11.534978 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-56654dcf7-bxzlp" Jul 23 11:46:38.550302 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.550260 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf"] Jul 23 11:46:38.550942 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.550924 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d304a614-c960-403e-8a87-9260701bd98a" containerName="authorino" Jul 23 11:46:38.550996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.550946 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d304a614-c960-403e-8a87-9260701bd98a" containerName="authorino" Jul 23 11:46:38.550996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.550970 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7efa7320-901f-4b1c-80f5-22f90a29953a" containerName="authorino" Jul 23 11:46:38.550996 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.550978 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="7efa7320-901f-4b1c-80f5-22f90a29953a" containerName="authorino" Jul 23 11:46:38.551128 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.551058 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d304a614-c960-403e-8a87-9260701bd98a" containerName="authorino" Jul 23 11:46:38.551128 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.551066 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="7efa7320-901f-4b1c-80f5-22f90a29953a" containerName="authorino" Jul 23 11:46:38.554225 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.554210 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.558091 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.558066 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 23 11:46:38.558091 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.558088 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-btpzj\"" Jul 23 11:46:38.558311 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.558131 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 23 11:46:38.558311 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.558140 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 23 11:46:38.575095 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.575073 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf"] Jul 23 11:46:38.685442 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.685413 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.685442 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.685446 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.685698 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.685484 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047434b9-606d-418f-a7e7-17dfebbcd8d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.685698 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.685560 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.685698 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.685596 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.685698 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.685615 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvl4\" (UniqueName: \"kubernetes.io/projected/047434b9-606d-418f-a7e7-17dfebbcd8d5-kube-api-access-ktvl4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787124 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787085 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787301 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787171 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047434b9-606d-418f-a7e7-17dfebbcd8d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787301 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787222 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787301 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787259 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787301 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787282 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvl4\" (UniqueName: \"kubernetes.io/projected/047434b9-606d-418f-a7e7-17dfebbcd8d5-kube-api-access-ktvl4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787512 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787398 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787571 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787544 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787727 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787707 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.787794 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.787750 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.789752 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.789728 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/047434b9-606d-418f-a7e7-17dfebbcd8d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.789854 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.789795 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/047434b9-606d-418f-a7e7-17dfebbcd8d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.795618 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.795597 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvl4\" (UniqueName: \"kubernetes.io/projected/047434b9-606d-418f-a7e7-17dfebbcd8d5-kube-api-access-ktvl4\") pod \"e2e-trlp-test-simulated-kserve-598466548d-8bsjf\" (UID: \"047434b9-606d-418f-a7e7-17dfebbcd8d5\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:38.865506 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:38.865416 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:39.006096 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:39.006073 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf"] Jul 23 11:46:39.008122 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:46:39.008098 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod047434b9_606d_418f_a7e7_17dfebbcd8d5.slice/crio-1ae107c216e6456e9f0944df0bce4e96a13b9078e4e8bb98c008a82774536e6b WatchSource:0}: Error finding container 1ae107c216e6456e9f0944df0bce4e96a13b9078e4e8bb98c008a82774536e6b: Status 404 returned error can't find the container with id 1ae107c216e6456e9f0944df0bce4e96a13b9078e4e8bb98c008a82774536e6b Jul 23 11:46:39.009574 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:39.009553 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 11:46:39.687816 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:39.687773 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" event={"ID":"047434b9-606d-418f-a7e7-17dfebbcd8d5","Type":"ContainerStarted","Data":"1ae107c216e6456e9f0944df0bce4e96a13b9078e4e8bb98c008a82774536e6b"} Jul 23 11:46:40.698277 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:40.698196 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" event={"ID":"047434b9-606d-418f-a7e7-17dfebbcd8d5","Type":"ContainerStarted","Data":"69728b1d9523d0375cf0fd92bfc5746b997755a5d52037a38ec9522196084421"} Jul 23 11:46:40.716907 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:40.716862 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" podStartSLOduration=1.690819273 podStartE2EDuration="2.716845024s" totalImagesPullingTime="1.026025751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:46:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:46:40.714836744 +0000 UTC m=+648.820163178" watchObservedRunningTime="2026-07-23 11:46:40.716845024 +0000 UTC m=+648.822171438" Jul 23 11:46:48.865908 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:48.865873 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:48.866378 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:48.865967 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:48.878007 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:48.877981 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:46:49.754795 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:46:49.754766 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-8bsjf" Jul 23 11:48:20.939986 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:20.939953 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-74c4f6c88d-gvthw"] Jul 23 11:48:20.943600 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:20.943583 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:20.952197 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:20.952170 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74c4f6c88d-gvthw"] Jul 23 11:48:21.041010 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.040983 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84bjh\" (UniqueName: \"kubernetes.io/projected/4b67f142-0f4c-43ca-830f-1f273a26050a-kube-api-access-84bjh\") pod \"authorino-74c4f6c88d-gvthw\" (UID: \"4b67f142-0f4c-43ca-830f-1f273a26050a\") " pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.041146 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.041044 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b67f142-0f4c-43ca-830f-1f273a26050a-tls-cert\") pod \"authorino-74c4f6c88d-gvthw\" (UID: \"4b67f142-0f4c-43ca-830f-1f273a26050a\") " pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.141460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.141432 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84bjh\" (UniqueName: \"kubernetes.io/projected/4b67f142-0f4c-43ca-830f-1f273a26050a-kube-api-access-84bjh\") pod \"authorino-74c4f6c88d-gvthw\" (UID: \"4b67f142-0f4c-43ca-830f-1f273a26050a\") " pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.141585 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.141483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b67f142-0f4c-43ca-830f-1f273a26050a-tls-cert\") pod \"authorino-74c4f6c88d-gvthw\" (UID: \"4b67f142-0f4c-43ca-830f-1f273a26050a\") " pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.143972 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.143953 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/4b67f142-0f4c-43ca-830f-1f273a26050a-tls-cert\") pod \"authorino-74c4f6c88d-gvthw\" (UID: \"4b67f142-0f4c-43ca-830f-1f273a26050a\") " pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.150518 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.150499 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84bjh\" (UniqueName: \"kubernetes.io/projected/4b67f142-0f4c-43ca-830f-1f273a26050a-kube-api-access-84bjh\") pod \"authorino-74c4f6c88d-gvthw\" (UID: \"4b67f142-0f4c-43ca-830f-1f273a26050a\") " pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.253340 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.253271 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-74c4f6c88d-gvthw" Jul 23 11:48:21.393369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:21.393337 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-74c4f6c88d-gvthw"] Jul 23 11:48:21.395501 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:48:21.395474 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b67f142_0f4c_43ca_830f_1f273a26050a.slice/crio-52f39233ac833b61b562259ba3176dcbe3712f5cebd12706dfd67e8e0d48f423 WatchSource:0}: Error finding container 52f39233ac833b61b562259ba3176dcbe3712f5cebd12706dfd67e8e0d48f423: Status 404 returned error can't find the container with id 52f39233ac833b61b562259ba3176dcbe3712f5cebd12706dfd67e8e0d48f423 Jul 23 11:48:22.202643 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.202531 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74c4f6c88d-gvthw" event={"ID":"4b67f142-0f4c-43ca-830f-1f273a26050a","Type":"ContainerStarted","Data":"7f7d975538a026320a04903df9e27f99b225d8ebc0f09cdd05a8662048429f22"} Jul 23 11:48:22.202643 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.202586 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-74c4f6c88d-gvthw" event={"ID":"4b67f142-0f4c-43ca-830f-1f273a26050a","Type":"ContainerStarted","Data":"52f39233ac833b61b562259ba3176dcbe3712f5cebd12706dfd67e8e0d48f423"} Jul 23 11:48:22.223203 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.223125 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-74c4f6c88d-gvthw" podStartSLOduration=1.786538394 podStartE2EDuration="2.223104098s" totalImagesPullingTime="436.565704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:48:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:48:22.221345097 +0000 UTC m=+750.326671510" watchObservedRunningTime="2026-07-23 11:48:22.223104098 +0000 UTC m=+750.328430703" Jul 23 11:48:22.260645 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.259937 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5679b76fd9-rmznc"] Jul 23 11:48:22.260645 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.260150 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5679b76fd9-rmznc" podUID="029bd78c-7e0f-4e30-a983-7b80da6e6bb5" containerName="authorino" containerID="cri-o://3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70" gracePeriod=30 Jul 23 11:48:22.543772 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.543744 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:48:22.655727 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.655690 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vdwvk\" (UniqueName: \"kubernetes.io/projected/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-kube-api-access-vdwvk\") pod \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " Jul 23 11:48:22.656021 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.656005 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-tls-cert\") pod \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\" (UID: \"029bd78c-7e0f-4e30-a983-7b80da6e6bb5\") " Jul 23 11:48:22.670216 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.670166 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-kube-api-access-vdwvk" (OuterVolumeSpecName: "kube-api-access-vdwvk") pod "029bd78c-7e0f-4e30-a983-7b80da6e6bb5" (UID: "029bd78c-7e0f-4e30-a983-7b80da6e6bb5"). InnerVolumeSpecName "kube-api-access-vdwvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:48:22.676928 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.676900 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "029bd78c-7e0f-4e30-a983-7b80da6e6bb5" (UID: "029bd78c-7e0f-4e30-a983-7b80da6e6bb5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 23 11:48:22.757034 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.757007 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdwvk\" (UniqueName: \"kubernetes.io/projected/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-kube-api-access-vdwvk\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:48:22.757034 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:22.757033 2554 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/029bd78c-7e0f-4e30-a983-7b80da6e6bb5-tls-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:48:23.209182 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.209150 2554 generic.go:356] "Generic (PLEG): container finished" podID="029bd78c-7e0f-4e30-a983-7b80da6e6bb5" containerID="3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70" exitCode=0 Jul 23 11:48:23.209637 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.209206 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5679b76fd9-rmznc" Jul 23 11:48:23.209637 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.209251 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5679b76fd9-rmznc" event={"ID":"029bd78c-7e0f-4e30-a983-7b80da6e6bb5","Type":"ContainerDied","Data":"3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70"} Jul 23 11:48:23.209637 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.209300 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5679b76fd9-rmznc" event={"ID":"029bd78c-7e0f-4e30-a983-7b80da6e6bb5","Type":"ContainerDied","Data":"1b1aa1882b857ddae8e047451fc3b43207f6975bd8130e8d2b701c97351647d8"} Jul 23 11:48:23.209637 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.209317 2554 scope.go:117] "RemoveContainer" containerID="3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70" Jul 23 11:48:23.218370 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.218353 2554 scope.go:117] "RemoveContainer" containerID="3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70" Jul 23 11:48:23.218669 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:48:23.218646 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70\": container with ID starting with 3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70 not found: ID does not exist" containerID="3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70" Jul 23 11:48:23.218734 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.218683 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70"} err="failed to get container status \"3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70\": rpc error: code = NotFound desc = could not find container \"3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70\": container with ID starting with 3ea29fa0a5cdc13ff103cadbe952e82f0ad7287ff7a014d4670c76bb1434ea70 not found: ID does not exist" Jul 23 11:48:23.240589 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.240563 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5679b76fd9-rmznc"] Jul 23 11:48:23.249173 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:23.249149 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5679b76fd9-rmznc"] Jul 23 11:48:24.453443 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:48:24.453410 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="029bd78c-7e0f-4e30-a983-7b80da6e6bb5" path="/var/lib/kubelet/pods/029bd78c-7e0f-4e30-a983-7b80da6e6bb5/volumes" Jul 23 11:55:13.383952 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:13.383857 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v"] Jul 23 11:55:13.384525 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:13.384101 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" podUID="5e744f37-fa15-45f3-9b6c-293d7570f93f" containerName="manager" containerID="cri-o://7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9" gracePeriod=10 Jul 23 11:55:16.737963 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:16.737936 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:55:16.901651 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:16.901547 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77plp\" (UniqueName: \"kubernetes.io/projected/5e744f37-fa15-45f3-9b6c-293d7570f93f-kube-api-access-77plp\") pod \"5e744f37-fa15-45f3-9b6c-293d7570f93f\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " Jul 23 11:55:16.901792 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:16.901669 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e744f37-fa15-45f3-9b6c-293d7570f93f-extensions-socket-volume\") pod \"5e744f37-fa15-45f3-9b6c-293d7570f93f\" (UID: \"5e744f37-fa15-45f3-9b6c-293d7570f93f\") " Jul 23 11:55:16.902009 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:16.901986 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e744f37-fa15-45f3-9b6c-293d7570f93f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5e744f37-fa15-45f3-9b6c-293d7570f93f" (UID: "5e744f37-fa15-45f3-9b6c-293d7570f93f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 11:55:16.903727 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:16.903700 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e744f37-fa15-45f3-9b6c-293d7570f93f-kube-api-access-77plp" (OuterVolumeSpecName: "kube-api-access-77plp") pod "5e744f37-fa15-45f3-9b6c-293d7570f93f" (UID: "5e744f37-fa15-45f3-9b6c-293d7570f93f"). InnerVolumeSpecName "kube-api-access-77plp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 11:55:17.002928 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.002901 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77plp\" (UniqueName: \"kubernetes.io/projected/5e744f37-fa15-45f3-9b6c-293d7570f93f-kube-api-access-77plp\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:55:17.002928 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.002926 2554 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5e744f37-fa15-45f3-9b6c-293d7570f93f-extensions-socket-volume\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 11:55:17.298917 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.298889 2554 generic.go:356] "Generic (PLEG): container finished" podID="5e744f37-fa15-45f3-9b6c-293d7570f93f" containerID="7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9" exitCode=0 Jul 23 11:55:17.299054 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.298935 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" Jul 23 11:55:17.299054 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.298935 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" event={"ID":"5e744f37-fa15-45f3-9b6c-293d7570f93f","Type":"ContainerDied","Data":"7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9"} Jul 23 11:55:17.299191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.299078 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v" event={"ID":"5e744f37-fa15-45f3-9b6c-293d7570f93f","Type":"ContainerDied","Data":"35dcc6df040d8dd86466bbc6a45b07d9514e24f2653653128f5a64632005c9c7"} Jul 23 11:55:17.299191 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.299121 2554 scope.go:117] "RemoveContainer" containerID="7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9" Jul 23 11:55:17.309423 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.309404 2554 scope.go:117] "RemoveContainer" containerID="7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9" Jul 23 11:55:17.309739 ip-10-0-141-203 kubenswrapper[2554]: E0723 11:55:17.309716 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9\": container with ID starting with 7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9 not found: ID does not exist" containerID="7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9" Jul 23 11:55:17.309801 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.309748 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9"} err="failed to get container status \"7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9\": rpc error: code = NotFound desc = could not find container \"7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9\": container with ID starting with 7306eb6f938413e3ebd0d7c65f97f3ca110d449cd25305909a67f7f8c2e6adf9 not found: ID does not exist" Jul 23 11:55:17.328237 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.328214 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v"] Jul 23 11:55:17.334502 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:17.334460 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-lgp6v"] Jul 23 11:55:18.453297 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:55:18.453261 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e744f37-fa15-45f3-9b6c-293d7570f93f" path="/var/lib/kubelet/pods/5e744f37-fa15-45f3-9b6c-293d7570f93f/volumes" Jul 23 11:56:14.821916 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.821878 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq"] Jul 23 11:56:14.822369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.822264 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="029bd78c-7e0f-4e30-a983-7b80da6e6bb5" containerName="authorino" Jul 23 11:56:14.822369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.822275 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="029bd78c-7e0f-4e30-a983-7b80da6e6bb5" containerName="authorino" Jul 23 11:56:14.822369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.822293 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e744f37-fa15-45f3-9b6c-293d7570f93f" containerName="manager" Jul 23 11:56:14.822369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.822298 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e744f37-fa15-45f3-9b6c-293d7570f93f" containerName="manager" Jul 23 11:56:14.822369 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.822363 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="029bd78c-7e0f-4e30-a983-7b80da6e6bb5" containerName="authorino" Jul 23 11:56:14.822530 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.822375 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e744f37-fa15-45f3-9b6c-293d7570f93f" containerName="manager" Jul 23 11:56:14.827146 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.827125 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:14.832732 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.832708 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-qddrk\"" Jul 23 11:56:14.889715 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.889690 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq"] Jul 23 11:56:14.983772 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.983745 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cca4d76-e5f6-466a-ab60-6aae804516c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ddcdq\" (UID: \"4cca4d76-e5f6-466a-ab60-6aae804516c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:14.983893 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:14.983791 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6dk4\" (UniqueName: \"kubernetes.io/projected/4cca4d76-e5f6-466a-ab60-6aae804516c6-kube-api-access-p6dk4\") pod \"kuadrant-operator-controller-manager-86878c769f-ddcdq\" (UID: \"4cca4d76-e5f6-466a-ab60-6aae804516c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.084664 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.084576 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6dk4\" (UniqueName: \"kubernetes.io/projected/4cca4d76-e5f6-466a-ab60-6aae804516c6-kube-api-access-p6dk4\") pod \"kuadrant-operator-controller-manager-86878c769f-ddcdq\" (UID: \"4cca4d76-e5f6-466a-ab60-6aae804516c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.084773 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.084669 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cca4d76-e5f6-466a-ab60-6aae804516c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ddcdq\" (UID: \"4cca4d76-e5f6-466a-ab60-6aae804516c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.084978 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.084962 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/4cca4d76-e5f6-466a-ab60-6aae804516c6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-ddcdq\" (UID: \"4cca4d76-e5f6-466a-ab60-6aae804516c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.095865 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.095844 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6dk4\" (UniqueName: \"kubernetes.io/projected/4cca4d76-e5f6-466a-ab60-6aae804516c6-kube-api-access-p6dk4\") pod \"kuadrant-operator-controller-manager-86878c769f-ddcdq\" (UID: \"4cca4d76-e5f6-466a-ab60-6aae804516c6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.137481 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.137459 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.323599 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.323578 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq"] Jul 23 11:56:15.326440 ip-10-0-141-203 kubenswrapper[2554]: W0723 11:56:15.326411 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cca4d76_e5f6_466a_ab60_6aae804516c6.slice/crio-c2696413a20fbe48ee6f5c783a7321cfcfc919cd63eb2a354a51f22bb00fd6ee WatchSource:0}: Error finding container c2696413a20fbe48ee6f5c783a7321cfcfc919cd63eb2a354a51f22bb00fd6ee: Status 404 returned error can't find the container with id c2696413a20fbe48ee6f5c783a7321cfcfc919cd63eb2a354a51f22bb00fd6ee Jul 23 11:56:15.589310 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.589272 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" event={"ID":"4cca4d76-e5f6-466a-ab60-6aae804516c6","Type":"ContainerStarted","Data":"6184eaac930323a13046bfd5329ba56d4e4123bb5ae025388b90e1dad11a2082"} Jul 23 11:56:15.589460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.589338 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" event={"ID":"4cca4d76-e5f6-466a-ab60-6aae804516c6","Type":"ContainerStarted","Data":"c2696413a20fbe48ee6f5c783a7321cfcfc919cd63eb2a354a51f22bb00fd6ee"} Jul 23 11:56:15.589460 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.589366 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 11:56:15.664547 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:15.664453 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" podStartSLOduration=1.664439309 podStartE2EDuration="1.664439309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 11:56:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 11:56:15.662372272 +0000 UTC m=+1223.767698685" watchObservedRunningTime="2026-07-23 11:56:15.664439309 +0000 UTC m=+1223.769765720" Jul 23 11:56:26.596398 ip-10-0-141-203 kubenswrapper[2554]: I0723 11:56:26.596363 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-ddcdq" Jul 23 12:03:44.200806 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.200777 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw"] Jul 23 12:03:44.204633 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.204601 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.209955 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.209936 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9fab72d8-openshift-default-dockercfg-t826g\"" Jul 23 12:03:44.263119 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.263096 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw"] Jul 23 12:03:44.294128 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294096 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-certs\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294313 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-podinfo\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294313 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294215 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-data\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294313 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294238 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-token\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294313 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294297 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-envoy\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294567 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294336 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-credential-socket\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294567 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294375 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istiod-ca-cert\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294567 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294404 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-socket\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.294567 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.294434 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfg4v\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-kube-api-access-sfg4v\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395091 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395063 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-certs\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395285 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395122 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-podinfo\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395285 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395171 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-data\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395285 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395193 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-token\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395285 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395229 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-envoy\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395285 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395269 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-credential-socket\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395553 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395308 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istiod-ca-cert\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395553 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395340 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-socket\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395553 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395371 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfg4v\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-kube-api-access-sfg4v\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395553 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395501 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-certs\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395795 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-data\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.395861 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.395839 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-socket\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.396048 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.396026 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-credential-socket\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.396146 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.396131 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istiod-ca-cert\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.397671 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.397648 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-envoy\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.397816 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.397797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-podinfo\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.410847 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.410819 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfg4v\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-kube-api-access-sfg4v\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.413025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.413006 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-token\") pod \"e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.515247 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.515178 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:44.656878 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.656854 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw"] Jul 23 12:03:44.661374 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:03:44.661300 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33857fb8_bec7_4eb4_9d8b_90e75e8d3a40.slice/crio-4a048f47fec88c8d33cbd408401113555bc4413c76debb5082c71dc5f745ac2b WatchSource:0}: Error finding container 4a048f47fec88c8d33cbd408401113555bc4413c76debb5082c71dc5f745ac2b: Status 404 returned error can't find the container with id 4a048f47fec88c8d33cbd408401113555bc4413c76debb5082c71dc5f745ac2b Jul 23 12:03:44.662936 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.662908 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:03:44.663015 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.662975 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:03:44.663015 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.663010 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:03:44.853903 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.853863 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" event={"ID":"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40","Type":"ContainerStarted","Data":"fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564"} Jul 23 12:03:44.854074 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.853940 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" event={"ID":"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40","Type":"ContainerStarted","Data":"4a048f47fec88c8d33cbd408401113555bc4413c76debb5082c71dc5f745ac2b"} Jul 23 12:03:44.877945 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:44.877895 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podStartSLOduration=0.877876402 podStartE2EDuration="877.876402ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:03:44.875984695 +0000 UTC m=+1672.981311106" watchObservedRunningTime="2026-07-23 12:03:44.877876402 +0000 UTC m=+1672.983202813" Jul 23 12:03:45.515430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:45.515388 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:45.516900 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:45.516873 2554 patch_prober.go:28] interesting pod/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 12:03:45.517013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:45.516930 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 12:03:46.523737 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:46.523706 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:46.865575 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:46.865542 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:46.866452 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:46.866434 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:57.939565 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:57.939452 2554 generic.go:356] "Generic (PLEG): container finished" podID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerID="fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564" exitCode=0 Jul 23 12:03:57.939565 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:57.939529 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" event={"ID":"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40","Type":"ContainerDied","Data":"fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564"} Jul 23 12:03:57.940087 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:57.939947 2554 scope.go:117] "RemoveContainer" containerID="fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564" Jul 23 12:03:57.941020 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:57.940992 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:03:57.941129 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:57.941100 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:03:57.941190 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:57.941150 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:03:58.946601 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:58.946567 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" event={"ID":"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40","Type":"ContainerStarted","Data":"a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea"} Jul 23 12:03:59.515569 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:59.515533 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:03:59.516758 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:59.516721 2554 patch_prober.go:28] interesting pod/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 12:03:59.516917 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:03:59.516791 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 12:04:00.515670 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:00.515608 2554 patch_prober.go:28] interesting pod/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 12:04:00.516122 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:00.515696 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 12:04:01.515727 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:01.515694 2554 patch_prober.go:28] interesting pod/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 12:04:01.516115 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:01.515753 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 12:04:01.865694 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:01.865653 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:04:02.516467 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:02.516429 2554 patch_prober.go:28] interesting pod/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 23 12:04:02.516846 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:02.516492 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 23 12:04:03.519480 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:03.519456 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:04:03.520471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:03.520455 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:04:27.264343 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:27.264267 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw"] Jul 23 12:04:27.264866 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:27.264498 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" containerID="cri-o://a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea" gracePeriod=30 Jul 23 12:04:28.404214 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.404181 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs"] Jul 23 12:04:28.407811 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.407789 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.410355 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.410327 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-9e27edaf-gw-openshift-default-dockercfg-cr52v\"" Jul 23 12:04:28.442440 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.442420 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs"] Jul 23 12:04:28.585875 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.585840 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istiod-ca-cert\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586065 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.585917 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-socket\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586065 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.585976 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-podinfo\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586065 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.586034 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-data\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586065 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.586061 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-envoy\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586240 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.586137 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-certs\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586240 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.586172 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-kube-api-access-552sq\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586240 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.586208 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-token\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.586240 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.586233 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-credential-socket\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.687716 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687637 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-socket\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.687716 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687680 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-podinfo\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.687716 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687702 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-data\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.687716 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687718 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-envoy\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687839 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-certs\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687873 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-kube-api-access-552sq\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687932 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-token\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.687968 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-credential-socket\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.688028 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istiod-ca-cert\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688292 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.688048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-data\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688292 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.688081 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-socket\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688292 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.688206 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-certs\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.688451 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.688384 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-credential-socket\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.689142 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.689117 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istiod-ca-cert\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.690160 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.690136 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-envoy\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.690418 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.690400 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-podinfo\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.707641 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.707602 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-token\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.708722 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.708695 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-kube-api-access-552sq\") pod \"e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.718341 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.718318 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:28.859805 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.859780 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs"] Jul 23 12:04:28.861086 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:04:28.861062 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d5a1b1_0f2e_4b03_aa7c_78d791b01e72.slice/crio-b0984532ba7e5fa7ddc727d3062bd6869cdd0a43858c905227b73284fae29440 WatchSource:0}: Error finding container b0984532ba7e5fa7ddc727d3062bd6869cdd0a43858c905227b73284fae29440: Status 404 returned error can't find the container with id b0984532ba7e5fa7ddc727d3062bd6869cdd0a43858c905227b73284fae29440 Jul 23 12:04:28.863213 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.863186 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:04:28.863287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.863251 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:04:28.863287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:28.863282 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:04:29.105142 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:29.105106 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" event={"ID":"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72","Type":"ContainerStarted","Data":"e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d"} Jul 23 12:04:29.105312 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:29.105160 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" event={"ID":"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72","Type":"ContainerStarted","Data":"b0984532ba7e5fa7ddc727d3062bd6869cdd0a43858c905227b73284fae29440"} Jul 23 12:04:29.146903 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:29.146849 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podStartSLOduration=1.146832374 podStartE2EDuration="1.146832374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:04:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:04:29.1433903 +0000 UTC m=+1717.248716713" watchObservedRunningTime="2026-07-23 12:04:29.146832374 +0000 UTC m=+1717.252158786" Jul 23 12:04:29.719111 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:29.719084 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:29.723878 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:29.723856 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:30.110557 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:30.110531 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:30.111322 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:30.111303 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:32.524802 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.524743 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:04:32.624445 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624417 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-certs\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624589 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624465 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-envoy\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624589 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624502 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-podinfo\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624589 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624576 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-credential-socket\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624739 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624668 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sfg4v\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-kube-api-access-sfg4v\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624789 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624714 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:32.624789 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624776 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istiod-ca-cert\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624868 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624796 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:32.624868 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624818 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-data\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624944 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624878 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-socket\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.624944 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.624919 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-token\") pod \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\" (UID: \"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40\") " Jul 23 12:04:32.625180 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625146 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:32.625180 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625169 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:04:32.625508 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625489 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.625556 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625516 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.625556 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625530 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.625556 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625545 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.625881 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.625859 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-data" (OuterVolumeSpecName: "istio-data") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:32.626881 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.626853 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:04:32.626977 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.626892 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:04:32.627412 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.627394 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-kube-api-access-sfg4v" (OuterVolumeSpecName: "kube-api-access-sfg4v") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "kube-api-access-sfg4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:04:32.627472 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.627439 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-token" (OuterVolumeSpecName: "istio-token") pod "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" (UID: "33857fb8-bec7-4eb4-9d8b-90e75e8d3a40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:04:32.726392 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.726373 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.726392 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.726393 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.726532 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.726401 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.726532 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.726408 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:32.726532 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:32.726416 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfg4v\" (UniqueName: \"kubernetes.io/projected/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40-kube-api-access-sfg4v\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:04:33.126070 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.126041 2554 generic.go:356] "Generic (PLEG): container finished" podID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerID="a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea" exitCode=0 Jul 23 12:04:33.126229 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.126101 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" Jul 23 12:04:33.126229 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.126137 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" event={"ID":"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40","Type":"ContainerDied","Data":"a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea"} Jul 23 12:04:33.126229 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.126222 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw" event={"ID":"33857fb8-bec7-4eb4-9d8b-90e75e8d3a40","Type":"ContainerDied","Data":"4a048f47fec88c8d33cbd408401113555bc4413c76debb5082c71dc5f745ac2b"} Jul 23 12:04:33.126418 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.126256 2554 scope.go:117] "RemoveContainer" containerID="a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea" Jul 23 12:04:33.135679 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.135661 2554 scope.go:117] "RemoveContainer" containerID="fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564" Jul 23 12:04:33.143456 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.143442 2554 scope.go:117] "RemoveContainer" containerID="a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea" Jul 23 12:04:33.143711 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:04:33.143694 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea\": container with ID starting with a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea not found: ID does not exist" containerID="a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea" Jul 23 12:04:33.143758 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.143719 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea"} err="failed to get container status \"a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea\": rpc error: code = NotFound desc = could not find container \"a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea\": container with ID starting with a6bd0b98cdd96bfce76ce5a0a5750a5d68d5e61ec223dbf67cafd9bd96666cea not found: ID does not exist" Jul 23 12:04:33.143758 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.143735 2554 scope.go:117] "RemoveContainer" containerID="fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564" Jul 23 12:04:33.143956 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:04:33.143938 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564\": container with ID starting with fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564 not found: ID does not exist" containerID="fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564" Jul 23 12:04:33.144011 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.143964 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564"} err="failed to get container status \"fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564\": rpc error: code = NotFound desc = could not find container \"fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564\": container with ID starting with fbdee09c3ec13d6d3a5248ff2607b69694974e23847b818b8064cc0947189564 not found: ID does not exist" Jul 23 12:04:33.160424 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.160397 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw"] Jul 23 12:04:33.167492 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:33.167472 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9fab72d8-openshift-default-75b476f777-lbsjw"] Jul 23 12:04:34.453984 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:34.453952 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" path="/var/lib/kubelet/pods/33857fb8-bec7-4eb4-9d8b-90e75e8d3a40/volumes" Jul 23 12:04:45.784392 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:45.783419 2554 patch_prober.go:28] interesting pod/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:46962->10.134.0.53:15021: read: connection reset by peer" start-of-body= Jul 23 12:04:45.784392 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:45.783519 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": read tcp 10.134.0.2:46962->10.134.0.53:15021: read: connection reset by peer" Jul 23 12:04:46.194366 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:46.194334 2554 generic.go:356] "Generic (PLEG): container finished" podID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerID="e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d" exitCode=0 Jul 23 12:04:46.194514 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:46.194408 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" event={"ID":"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72","Type":"ContainerDied","Data":"e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d"} Jul 23 12:04:46.194854 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:46.194838 2554 scope.go:117] "RemoveContainer" containerID="e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d" Jul 23 12:04:46.195860 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:46.195831 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:04:46.195931 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:46.195918 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:04:46.195982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:46.195967 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:04:47.206396 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:47.206350 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" event={"ID":"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72","Type":"ContainerStarted","Data":"a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac"} Jul 23 12:04:47.719208 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:47.719174 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:47.719381 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:47.719364 2554 patch_prober.go:28] interesting pod/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 23 12:04:47.719431 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:47.719412 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 23 12:04:48.719356 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:48.719322 2554 patch_prober.go:28] interesting pod/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 23 12:04:48.719759 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:48.719383 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 23 12:04:49.718898 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:49.718852 2554 patch_prober.go:28] interesting pod/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 23 12:04:49.719079 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:49.718916 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 23 12:04:50.719075 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:50.719042 2554 patch_prober.go:28] interesting pod/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body= Jul 23 12:04:50.719442 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:50.719091 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" Jul 23 12:04:51.722581 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:51.722547 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:51.723017 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:51.722992 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:04:51.723782 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:04:51.723765 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:05:10.988261 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:10.988231 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs"] Jul 23 12:05:10.988797 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:10.988443 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" containerID="cri-o://a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac" gracePeriod=30 Jul 23 12:05:11.884060 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884023 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk"] Jul 23 12:05:11.884513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884495 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" Jul 23 12:05:11.884513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884514 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" Jul 23 12:05:11.884662 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884522 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" Jul 23 12:05:11.884662 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884529 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" Jul 23 12:05:11.884662 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884598 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" Jul 23 12:05:11.884662 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.884606 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="33857fb8-bec7-4eb4-9d8b-90e75e8d3a40" containerName="istio-proxy" Jul 23 12:05:11.887815 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.887796 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:11.890887 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.890844 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c1664359-openshift-default-dockercfg-tgzrc\"" Jul 23 12:05:11.906163 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:11.906108 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk"] Jul 23 12:05:12.064496 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064464 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-envoy\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064512 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-socket\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064537 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-data\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064616 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-credential-socket\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064700 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed089633-3894-4db3-b635-85e28d93cea4-istiod-ca-cert\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064729 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-istio-token\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064769 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29546\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-kube-api-access-29546\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064811 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-certs\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.064892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.064833 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed089633-3894-4db3-b635-85e28d93cea4-istio-podinfo\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.165982 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29546\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-kube-api-access-29546\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166040 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-certs\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166063 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed089633-3894-4db3-b635-85e28d93cea4-istio-podinfo\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166098 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-envoy\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166145 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-socket\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166183 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-data\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166207 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-credential-socket\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166259 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed089633-3894-4db3-b635-85e28d93cea4-istiod-ca-cert\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166295 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-istio-token\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166712 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166509 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-certs\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166712 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166604 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-credential-socket\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166712 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166663 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-data\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166865 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166721 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-socket\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.166997 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.166934 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed089633-3894-4db3-b635-85e28d93cea4-istiod-ca-cert\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.168587 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.168562 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-envoy\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.168812 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.168797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed089633-3894-4db3-b635-85e28d93cea4-istio-podinfo\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.176012 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.175986 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29546\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-kube-api-access-29546\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.176316 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.176290 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-istio-token\") pod \"e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.197857 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.197831 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:12.359310 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.359280 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk"] Jul 23 12:05:12.360588 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:05:12.360554 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded089633_3894_4db3_b635_85e28d93cea4.slice/crio-d0a1bcc5120124097872c94dcde9f64c967e8905e123bc194bde1bb8dcd5d628 WatchSource:0}: Error finding container d0a1bcc5120124097872c94dcde9f64c967e8905e123bc194bde1bb8dcd5d628: Status 404 returned error can't find the container with id d0a1bcc5120124097872c94dcde9f64c967e8905e123bc194bde1bb8dcd5d628 Jul 23 12:05:12.363030 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.362993 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:12.363130 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.363077 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:12.363130 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:12.363115 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:13.346290 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:13.346247 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" event={"ID":"ed089633-3894-4db3-b635-85e28d93cea4","Type":"ContainerStarted","Data":"0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164"} Jul 23 12:05:13.346782 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:13.346346 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" event={"ID":"ed089633-3894-4db3-b635-85e28d93cea4","Type":"ContainerStarted","Data":"d0a1bcc5120124097872c94dcde9f64c967e8905e123bc194bde1bb8dcd5d628"} Jul 23 12:05:13.382448 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:13.382382 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podStartSLOduration=2.382361114 podStartE2EDuration="2.382361114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:05:13.380477409 +0000 UTC m=+1761.485803821" watchObservedRunningTime="2026-07-23 12:05:13.382361114 +0000 UTC m=+1761.487687528" Jul 23 12:05:14.198598 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:14.198556 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:14.203681 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:14.203656 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:14.351253 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:14.351213 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:14.352084 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:14.352062 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:16.243591 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.243564 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:05:16.362790 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.362754 2554 generic.go:356] "Generic (PLEG): container finished" podID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerID="a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac" exitCode=0 Jul 23 12:05:16.362982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.362806 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" Jul 23 12:05:16.362982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.362830 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" event={"ID":"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72","Type":"ContainerDied","Data":"a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac"} Jul 23 12:05:16.362982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.362885 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs" event={"ID":"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72","Type":"ContainerDied","Data":"b0984532ba7e5fa7ddc727d3062bd6869cdd0a43858c905227b73284fae29440"} Jul 23 12:05:16.362982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.362907 2554 scope.go:117] "RemoveContainer" containerID="a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac" Jul 23 12:05:16.371820 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.371804 2554 scope.go:117] "RemoveContainer" containerID="e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d" Jul 23 12:05:16.379736 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.379704 2554 scope.go:117] "RemoveContainer" containerID="a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac" Jul 23 12:05:16.380050 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:05:16.380011 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac\": container with ID starting with a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac not found: ID does not exist" containerID="a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac" Jul 23 12:05:16.380131 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.380060 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac"} err="failed to get container status \"a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac\": rpc error: code = NotFound desc = could not find container \"a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac\": container with ID starting with a0355639f8b61d019853ff8720ee5b4113a140500240409f4690f066826babac not found: ID does not exist" Jul 23 12:05:16.380131 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.380084 2554 scope.go:117] "RemoveContainer" containerID="e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d" Jul 23 12:05:16.380381 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:05:16.380359 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d\": container with ID starting with e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d not found: ID does not exist" containerID="e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d" Jul 23 12:05:16.380433 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.380390 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d"} err="failed to get container status \"e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d\": rpc error: code = NotFound desc = could not find container \"e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d\": container with ID starting with e4280b8b6ae6b84bdd92f4754f276975a98a0207f19f8d9df3a0e9a69f17cc0d not found: ID does not exist" Jul 23 12:05:16.410323 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410286 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-certs\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410495 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410363 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istiod-ca-cert\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410495 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410389 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-podinfo\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410495 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410410 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-credential-socket\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410495 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410437 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-token\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410495 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410471 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-data\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410803 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410512 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-kube-api-access-552sq\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410803 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410560 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-envoy\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410803 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410556 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:05:16.410803 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410653 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-socket\") pod \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\" (UID: \"52d5a1b1-0f2e-4b03-aa7c-78d791b01e72\") " Jul 23 12:05:16.410803 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410733 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:05:16.411077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410802 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:05:16.411077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.410993 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:05:16.411077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.411009 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.411077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.411027 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.411077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.411037 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.411738 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.411709 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-data" (OuterVolumeSpecName: "istio-data") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:05:16.412917 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.412871 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:05:16.413017 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.412909 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-kube-api-access-552sq" (OuterVolumeSpecName: "kube-api-access-552sq") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "kube-api-access-552sq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:05:16.413017 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.412963 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-token" (OuterVolumeSpecName: "istio-token") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:05:16.413017 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.412990 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" (UID: "52d5a1b1-0f2e-4b03-aa7c-78d791b01e72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:05:16.511598 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.511560 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.511598 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.511597 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.511804 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.511611 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.511804 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.511654 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.511804 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.511668 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-552sq\" (UniqueName: \"kubernetes.io/projected/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-kube-api-access-552sq\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.511804 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.511682 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:05:16.687551 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.687465 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs"] Jul 23 12:05:16.690594 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:16.690566 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-9e27edaf-gw-openshift-default-6745bbbddc-2j6vs"] Jul 23 12:05:18.453869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:18.453834 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" path="/var/lib/kubelet/pods/52d5a1b1-0f2e-4b03-aa7c-78d791b01e72/volumes" Jul 23 12:05:29.352815 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.352777 2554 patch_prober.go:28] interesting pod/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 23 12:05:29.353207 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.352835 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 23 12:05:29.441582 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.441551 2554 generic.go:356] "Generic (PLEG): container finished" podID="ed089633-3894-4db3-b635-85e28d93cea4" containerID="0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164" exitCode=0 Jul 23 12:05:29.441728 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.441601 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" event={"ID":"ed089633-3894-4db3-b635-85e28d93cea4","Type":"ContainerDied","Data":"0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164"} Jul 23 12:05:29.441983 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.441957 2554 scope.go:117] "RemoveContainer" containerID="0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164" Jul 23 12:05:29.443019 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.442994 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:29.443077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.443058 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:29.443113 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:29.443087 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:30.459551 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:30.459514 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" event={"ID":"ed089633-3894-4db3-b635-85e28d93cea4","Type":"ContainerStarted","Data":"2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d"} Jul 23 12:05:31.198864 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:31.198776 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:31.198864 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:31.198821 2554 patch_prober.go:28] interesting pod/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 23 12:05:31.199148 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:31.198878 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 23 12:05:32.198961 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:32.198921 2554 patch_prober.go:28] interesting pod/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 23 12:05:32.199513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:32.199481 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 23 12:05:33.198819 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:33.198782 2554 patch_prober.go:28] interesting pod/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 23 12:05:33.199107 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:33.199077 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 23 12:05:34.198530 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:34.198496 2554 patch_prober.go:28] interesting pod/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Jul 23 12:05:34.198846 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:34.198816 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Jul 23 12:05:35.201909 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:35.201874 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:35.202280 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:35.202264 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:35.203162 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:35.203137 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:05:57.157205 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.157158 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk"] Jul 23 12:05:57.157695 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.157387 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" containerID="cri-o://2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d" gracePeriod=30 Jul 23 12:05:57.993857 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.993820 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d"] Jul 23 12:05:57.994259 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.994248 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" Jul 23 12:05:57.994309 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.994261 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" Jul 23 12:05:57.994309 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.994276 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" Jul 23 12:05:57.994309 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.994282 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" Jul 23 12:05:57.994437 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.994347 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" Jul 23 12:05:57.998131 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:57.998112 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.001361 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.001343 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c2bbf915-openshift-default-dockercfg-jkbdb\"" Jul 23 12:05:58.015518 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.015498 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d"] Jul 23 12:05:58.084242 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084202 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-podinfo\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084273 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-socket\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084382 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-token\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084568 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084472 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-data\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084705 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084561 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-credential-socket\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084705 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084602 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-certs\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084705 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084686 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-envoy\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084733 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istiod-ca-cert\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.084873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.084758 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xznxk\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-kube-api-access-xznxk\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.185900 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.185861 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-token\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.185912 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-data\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.185966 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-credential-socket\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.185998 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-certs\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186053 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-envoy\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186095 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istiod-ca-cert\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186118 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xznxk\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-kube-api-access-xznxk\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186157 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-podinfo\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186362 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186190 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-socket\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186753 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186373 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-data\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186753 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186391 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-certs\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186753 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186502 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-socket\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.186753 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.186666 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-credential-socket\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.187241 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.187221 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istiod-ca-cert\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.188585 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.188563 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-envoy\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.188803 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.188787 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-podinfo\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.194463 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.194433 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-token\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.194558 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.194520 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xznxk\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-kube-api-access-xznxk\") pod \"e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.308329 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.308296 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:58.456964 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.456940 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d"] Jul 23 12:05:58.459069 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:05:58.459044 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eac4e47_7550_4963_9cc5_7350f0b1cb58.slice/crio-94d73d449e64817d71624c455d0e7a8f42a07722689256384e8b99abcc87a2f4 WatchSource:0}: Error finding container 94d73d449e64817d71624c455d0e7a8f42a07722689256384e8b99abcc87a2f4: Status 404 returned error can't find the container with id 94d73d449e64817d71624c455d0e7a8f42a07722689256384e8b99abcc87a2f4 Jul 23 12:05:58.461209 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.461173 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:58.461293 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.461261 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:58.461333 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.461311 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:05:58.593704 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.593583 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" event={"ID":"3eac4e47-7550-4963-9cc5-7350f0b1cb58","Type":"ContainerStarted","Data":"e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370"} Jul 23 12:05:58.593704 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.593694 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" event={"ID":"3eac4e47-7550-4963-9cc5-7350f0b1cb58","Type":"ContainerStarted","Data":"94d73d449e64817d71624c455d0e7a8f42a07722689256384e8b99abcc87a2f4"} Jul 23 12:05:58.620835 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:58.620770 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podStartSLOduration=1.620747792 podStartE2EDuration="1.620747792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:05:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:05:58.620351622 +0000 UTC m=+1806.725678054" watchObservedRunningTime="2026-07-23 12:05:58.620747792 +0000 UTC m=+1806.726074208" Jul 23 12:05:59.309108 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:59.309072 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:59.314007 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:59.313983 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:59.598708 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:59.598599 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:05:59.599612 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:05:59.599593 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:02.418241 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.418188 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:06:02.530576 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530548 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-certs\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530736 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530607 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-envoy\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530736 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530673 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed089633-3894-4db3-b635-85e28d93cea4-istio-podinfo\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530736 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530710 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-istio-token\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530736 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530735 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-29546\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-kube-api-access-29546\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530776 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed089633-3894-4db3-b635-85e28d93cea4-istiod-ca-cert\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530804 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-data\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530834 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-credential-socket\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.530969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530870 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:02.530969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.530897 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-socket\") pod \"ed089633-3894-4db3-b635-85e28d93cea4\" (UID: \"ed089633-3894-4db3-b635-85e28d93cea4\") " Jul 23 12:06:02.531281 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.531190 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:02.531281 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.531266 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.531549 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.531404 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:02.531606 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.531576 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed089633-3894-4db3-b635-85e28d93cea4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:06:02.531980 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.531955 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-data" (OuterVolumeSpecName: "istio-data") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:02.532976 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.532953 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-istio-token" (OuterVolumeSpecName: "istio-token") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:06:02.533074 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.533029 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed089633-3894-4db3-b635-85e28d93cea4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:06:02.533074 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.533056 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:02.533162 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.533096 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-kube-api-access-29546" (OuterVolumeSpecName: "kube-api-access-29546") pod "ed089633-3894-4db3-b635-85e28d93cea4" (UID: "ed089633-3894-4db3-b635-85e28d93cea4"). InnerVolumeSpecName "kube-api-access-29546". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:06:02.615478 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.615450 2554 generic.go:356] "Generic (PLEG): container finished" podID="ed089633-3894-4db3-b635-85e28d93cea4" containerID="2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d" exitCode=0 Jul 23 12:06:02.615582 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.615479 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" event={"ID":"ed089633-3894-4db3-b635-85e28d93cea4","Type":"ContainerDied","Data":"2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d"} Jul 23 12:06:02.615582 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.615502 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" Jul 23 12:06:02.615582 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.615535 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk" event={"ID":"ed089633-3894-4db3-b635-85e28d93cea4","Type":"ContainerDied","Data":"d0a1bcc5120124097872c94dcde9f64c967e8905e123bc194bde1bb8dcd5d628"} Jul 23 12:06:02.615582 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.615558 2554 scope.go:117] "RemoveContainer" containerID="2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d" Jul 23 12:06:02.625360 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.625343 2554 scope.go:117] "RemoveContainer" containerID="0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164" Jul 23 12:06:02.631925 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631906 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.631925 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631926 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed089633-3894-4db3-b635-85e28d93cea4-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.632033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631934 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.632033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631943 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29546\" (UniqueName: \"kubernetes.io/projected/ed089633-3894-4db3-b635-85e28d93cea4-kube-api-access-29546\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.632033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631951 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed089633-3894-4db3-b635-85e28d93cea4-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.632033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631959 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.632033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631967 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.632033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.631976 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed089633-3894-4db3-b635-85e28d93cea4-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:02.633741 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.633729 2554 scope.go:117] "RemoveContainer" containerID="2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d" Jul 23 12:06:02.633971 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:06:02.633957 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d\": container with ID starting with 2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d not found: ID does not exist" containerID="2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d" Jul 23 12:06:02.634010 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.633979 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d"} err="failed to get container status \"2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d\": rpc error: code = NotFound desc = could not find container \"2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d\": container with ID starting with 2c90f1edb5906fd41a2c763f3e6e411a000ad5b82070c2ceb9549d594ac1183d not found: ID does not exist" Jul 23 12:06:02.634010 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.633994 2554 scope.go:117] "RemoveContainer" containerID="0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164" Jul 23 12:06:02.634224 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:06:02.634209 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164\": container with ID starting with 0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164 not found: ID does not exist" containerID="0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164" Jul 23 12:06:02.634276 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.634227 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164"} err="failed to get container status \"0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164\": rpc error: code = NotFound desc = could not find container \"0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164\": container with ID starting with 0e8ecb9121e4032e556b8b1112b160cd255671d503a41bb5a9cac7143c221164 not found: ID does not exist" Jul 23 12:06:02.640008 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.639986 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk"] Jul 23 12:06:02.643750 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:02.643730 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c1664359-openshift-default-5ff5d6cb8f-jfrsk"] Jul 23 12:06:04.453896 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:04.453862 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed089633-3894-4db3-b635-85e28d93cea4" path="/var/lib/kubelet/pods/ed089633-3894-4db3-b635-85e28d93cea4/volumes" Jul 23 12:06:14.664076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:14.663568 2554 patch_prober.go:28] interesting pod/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:40446->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 23 12:06:14.664076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:14.663663 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:40446->10.134.0.55:15021: read: connection reset by peer" Jul 23 12:06:15.681467 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:15.681435 2554 generic.go:356] "Generic (PLEG): container finished" podID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerID="e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370" exitCode=0 Jul 23 12:06:15.681871 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:15.681481 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" event={"ID":"3eac4e47-7550-4963-9cc5-7350f0b1cb58","Type":"ContainerDied","Data":"e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370"} Jul 23 12:06:15.681871 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:15.681858 2554 scope.go:117] "RemoveContainer" containerID="e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370" Jul 23 12:06:15.683612 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:15.683573 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:06:15.683937 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:15.683911 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:06:15.684125 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:15.684107 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:06:16.695074 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:16.694108 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" event={"ID":"3eac4e47-7550-4963-9cc5-7350f0b1cb58","Type":"ContainerStarted","Data":"3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583"} Jul 23 12:06:17.308721 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:17.308676 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:17.308991 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:17.308962 2554 patch_prober.go:28] interesting pod/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 23 12:06:17.309065 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:17.309045 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 23 12:06:18.308968 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:18.308929 2554 patch_prober.go:28] interesting pod/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 23 12:06:18.309355 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:18.308983 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 23 12:06:19.308928 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:19.308887 2554 patch_prober.go:28] interesting pod/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 23 12:06:19.309134 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:19.309001 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 23 12:06:20.309583 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:20.309544 2554 patch_prober.go:28] interesting pod/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 23 12:06:20.310008 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:20.309650 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 23 12:06:21.312592 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:21.312561 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:21.313076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:21.312997 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:21.313760 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:21.313742 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:40.300274 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:40.300231 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d"] Jul 23 12:06:40.300882 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:40.300527 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" containerID="cri-o://3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583" gracePeriod=30 Jul 23 12:06:45.589739 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.589710 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:45.644838 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644815 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-podinfo\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.644974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644852 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istiod-ca-cert\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.644974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644877 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-data\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.644974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644899 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-certs\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.644974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644924 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xznxk\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-kube-api-access-xznxk\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.644974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644955 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-envoy\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.645228 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.644977 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-socket\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.645228 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645030 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-token\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.645228 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645063 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-credential-socket\") pod \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\" (UID: \"3eac4e47-7550-4963-9cc5-7350f0b1cb58\") " Jul 23 12:06:45.645228 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645194 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:45.645419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645196 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:06:45.645419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645342 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:45.645419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645356 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:45.645577 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645458 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.645577 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645478 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.645577 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645491 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.645577 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.645503 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.646324 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.646298 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-data" (OuterVolumeSpecName: "istio-data") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:45.647421 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.647389 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-token" (OuterVolumeSpecName: "istio-token") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:06:45.647521 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.647461 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:06:45.647521 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.647465 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:06:45.647741 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.647717 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-kube-api-access-xznxk" (OuterVolumeSpecName: "kube-api-access-xznxk") pod "3eac4e47-7550-4963-9cc5-7350f0b1cb58" (UID: "3eac4e47-7550-4963-9cc5-7350f0b1cb58"). InnerVolumeSpecName "kube-api-access-xznxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:06:45.745999 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.745971 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xznxk\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-kube-api-access-xznxk\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.745999 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.745999 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.746158 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.746008 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.746158 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.746016 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.746158 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.746024 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3eac4e47-7550-4963-9cc5-7350f0b1cb58-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:06:45.843693 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.843582 2554 generic.go:356] "Generic (PLEG): container finished" podID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerID="3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583" exitCode=0 Jul 23 12:06:45.843693 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.843680 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" Jul 23 12:06:45.843902 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.843674 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" event={"ID":"3eac4e47-7550-4963-9cc5-7350f0b1cb58","Type":"ContainerDied","Data":"3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583"} Jul 23 12:06:45.843902 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.843812 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d" event={"ID":"3eac4e47-7550-4963-9cc5-7350f0b1cb58","Type":"ContainerDied","Data":"94d73d449e64817d71624c455d0e7a8f42a07722689256384e8b99abcc87a2f4"} Jul 23 12:06:45.843902 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.843842 2554 scope.go:117] "RemoveContainer" containerID="3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583" Jul 23 12:06:45.854269 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.854245 2554 scope.go:117] "RemoveContainer" containerID="e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370" Jul 23 12:06:45.862851 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.862833 2554 scope.go:117] "RemoveContainer" containerID="3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583" Jul 23 12:06:45.863153 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:06:45.863137 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583\": container with ID starting with 3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583 not found: ID does not exist" containerID="3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583" Jul 23 12:06:45.863229 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.863160 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583"} err="failed to get container status \"3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583\": rpc error: code = NotFound desc = could not find container \"3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583\": container with ID starting with 3bf546a3929f0001cc837ba4602ced9eb93b81a34d294d4978d075e86bc77583 not found: ID does not exist" Jul 23 12:06:45.863229 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.863176 2554 scope.go:117] "RemoveContainer" containerID="e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370" Jul 23 12:06:45.863442 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:06:45.863412 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370\": container with ID starting with e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370 not found: ID does not exist" containerID="e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370" Jul 23 12:06:45.863489 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.863447 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370"} err="failed to get container status \"e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370\": rpc error: code = NotFound desc = could not find container \"e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370\": container with ID starting with e55b53f4452d901857ced23e3cc5451faa738b21e35094fc93b53ee4493e1370 not found: ID does not exist" Jul 23 12:06:45.867468 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.867446 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d"] Jul 23 12:06:45.871555 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:45.871533 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c2bbf915-openshift-default-764d44cbc9-65f4d"] Jul 23 12:06:46.454064 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:06:46.454027 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" path="/var/lib/kubelet/pods/3eac4e47-7550-4963-9cc5-7350f0b1cb58/volumes" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.837165 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj"] Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838039 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838061 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838076 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838085 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838148 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838157 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838327 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838341 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838356 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="52d5a1b1-0f2e-4b03-aa7c-78d791b01e72" containerName="istio-proxy" Jul 23 12:08:23.840088 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.838371 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" Jul 23 12:08:23.842232 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.842210 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.846644 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.846610 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-110e698d-openshift-default-dockercfg-k996n\"" Jul 23 12:08:23.868216 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.868188 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj"] Jul 23 12:08:23.987572 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987541 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-data\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.987784 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987580 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-envoy\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.987784 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987606 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4smd\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-kube-api-access-j4smd\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.987784 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987650 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74580157-9acd-488b-b644-2af27f13f6c4-istiod-ca-cert\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.987784 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987685 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-socket\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.987784 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987752 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-credential-socket\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.988012 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987793 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-certs\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.988012 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987824 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-istio-token\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:23.988012 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:23.987865 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74580157-9acd-488b-b644-2af27f13f6c4-istio-podinfo\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.088783 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088746 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4smd\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-kube-api-access-j4smd\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.088783 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088781 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74580157-9acd-488b-b644-2af27f13f6c4-istiod-ca-cert\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088809 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-socket\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088838 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-credential-socket\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088864 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-certs\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088887 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-istio-token\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.088922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74580157-9acd-488b-b644-2af27f13f6c4-istio-podinfo\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089025 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089006 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-data\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089315 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089038 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-envoy\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089315 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089229 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-socket\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089420 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089331 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-certs\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089420 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089389 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-credential-socket\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089528 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089472 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-data\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.089794 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.089773 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74580157-9acd-488b-b644-2af27f13f6c4-istiod-ca-cert\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.091491 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.091427 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-envoy\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.091579 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.091557 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74580157-9acd-488b-b644-2af27f13f6c4-istio-podinfo\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.100267 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.100239 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4smd\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-kube-api-access-j4smd\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.100377 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.100301 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-istio-token\") pod \"e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.153280 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.153248 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:24.299145 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.299118 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj"] Jul 23 12:08:24.300859 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:08:24.300832 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74580157_9acd_488b_b644_2af27f13f6c4.slice/crio-1859d83449abce47f797a29d95b10b8362965be2a790023a9404a6ca199c189b WatchSource:0}: Error finding container 1859d83449abce47f797a29d95b10b8362965be2a790023a9404a6ca199c189b: Status 404 returned error can't find the container with id 1859d83449abce47f797a29d95b10b8362965be2a790023a9404a6ca199c189b Jul 23 12:08:24.303403 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.303373 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:08:24.303464 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.303448 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:08:24.303505 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.303476 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:08:24.370592 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.370565 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" event={"ID":"74580157-9acd-488b-b644-2af27f13f6c4","Type":"ContainerStarted","Data":"93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6"} Jul 23 12:08:24.370741 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.370613 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" event={"ID":"74580157-9acd-488b-b644-2af27f13f6c4","Type":"ContainerStarted","Data":"1859d83449abce47f797a29d95b10b8362965be2a790023a9404a6ca199c189b"} Jul 23 12:08:24.396594 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:24.396541 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podStartSLOduration=1.39652493 podStartE2EDuration="1.39652493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:08:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:08:24.395159899 +0000 UTC m=+1952.500486313" watchObservedRunningTime="2026-07-23 12:08:24.39652493 +0000 UTC m=+1952.501851395" Jul 23 12:08:25.154268 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:25.154239 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:25.159209 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:25.159179 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:25.376742 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:25.376716 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:25.377651 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:25.377608 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:40.462530 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:40.462384 2554 patch_prober.go:28] interesting pod/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:56608->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 23 12:08:40.462530 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:40.462456 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:56608->10.134.0.56:15021: read: connection reset by peer" Jul 23 12:08:41.461774 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:41.461735 2554 generic.go:356] "Generic (PLEG): container finished" podID="74580157-9acd-488b-b644-2af27f13f6c4" containerID="93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6" exitCode=0 Jul 23 12:08:41.461945 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:41.461795 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" event={"ID":"74580157-9acd-488b-b644-2af27f13f6c4","Type":"ContainerDied","Data":"93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6"} Jul 23 12:08:41.462215 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:41.462202 2554 scope.go:117] "RemoveContainer" containerID="93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6" Jul 23 12:08:41.463200 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:41.463174 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:08:41.463533 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:41.463253 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:08:41.463533 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:41.463303 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:08:42.472747 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:42.471710 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" event={"ID":"74580157-9acd-488b-b644-2af27f13f6c4","Type":"ContainerStarted","Data":"b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30"} Jul 23 12:08:43.153690 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:43.153658 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:43.153923 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:43.153900 2554 patch_prober.go:28] interesting pod/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 23 12:08:43.153971 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:43.153957 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 23 12:08:44.154603 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:44.154554 2554 patch_prober.go:28] interesting pod/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 23 12:08:44.155036 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:44.154657 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 23 12:08:45.154374 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:45.154329 2554 patch_prober.go:28] interesting pod/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 23 12:08:45.154597 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:45.154412 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 23 12:08:46.154587 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:46.154550 2554 patch_prober.go:28] interesting pod/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 23 12:08:46.154990 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:46.154602 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 23 12:08:47.158182 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:47.158148 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:47.158721 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:47.158538 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:08:47.159471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:08:47.159450 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:09:01.198459 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:01.198414 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj"] Jul 23 12:09:01.199028 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:01.198693 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" containerID="cri-o://b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30" gracePeriod=30 Jul 23 12:09:02.160566 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:02.160531 2554 patch_prober.go:28] interesting pod/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:09:02.160761 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:02.160597 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:09:06.454796 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.454772 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:09:06.596067 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.596041 2554 generic.go:356] "Generic (PLEG): container finished" podID="74580157-9acd-488b-b644-2af27f13f6c4" containerID="b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30" exitCode=0 Jul 23 12:09:06.596199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.596088 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" Jul 23 12:09:06.596199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.596117 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" event={"ID":"74580157-9acd-488b-b644-2af27f13f6c4","Type":"ContainerDied","Data":"b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30"} Jul 23 12:09:06.596199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.596168 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj" event={"ID":"74580157-9acd-488b-b644-2af27f13f6c4","Type":"ContainerDied","Data":"1859d83449abce47f797a29d95b10b8362965be2a790023a9404a6ca199c189b"} Jul 23 12:09:06.596199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.596189 2554 scope.go:117] "RemoveContainer" containerID="b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30" Jul 23 12:09:06.605151 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.605132 2554 scope.go:117] "RemoveContainer" containerID="93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6" Jul 23 12:09:06.608270 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608251 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-credential-socket\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608372 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608299 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74580157-9acd-488b-b644-2af27f13f6c4-istio-podinfo\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608372 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608340 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-istio-token\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608372 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608365 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-data\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608545 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608394 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-certs\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608545 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608456 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74580157-9acd-488b-b644-2af27f13f6c4-istiod-ca-cert\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608545 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608516 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-socket\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608721 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608553 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-envoy\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608721 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608584 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4smd\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-kube-api-access-j4smd\") pod \"74580157-9acd-488b-b644-2af27f13f6c4\" (UID: \"74580157-9acd-488b-b644-2af27f13f6c4\") " Jul 23 12:09:06.608987 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608494 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:06.608987 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608669 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:06.608987 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608806 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:06.608987 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.608828 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74580157-9acd-488b-b644-2af27f13f6c4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:09:06.609502 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.609432 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-data" (OuterVolumeSpecName: "istio-data") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:06.610722 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.610701 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/74580157-9acd-488b-b644-2af27f13f6c4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:09:06.611014 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.610986 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-kube-api-access-j4smd" (OuterVolumeSpecName: "kube-api-access-j4smd") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "kube-api-access-j4smd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:09:06.611198 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.611175 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:09:06.611198 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.611192 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-istio-token" (OuterVolumeSpecName: "istio-token") pod "74580157-9acd-488b-b644-2af27f13f6c4" (UID: "74580157-9acd-488b-b644-2af27f13f6c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:09:06.618335 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.616922 2554 scope.go:117] "RemoveContainer" containerID="b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30" Jul 23 12:09:06.618335 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:09:06.618320 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30\": container with ID starting with b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30 not found: ID does not exist" containerID="b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30" Jul 23 12:09:06.618437 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.618345 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30"} err="failed to get container status \"b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30\": rpc error: code = NotFound desc = could not find container \"b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30\": container with ID starting with b63365bb4bfee622b90454c56003cb8bc8d5ca02c0d6a9ce29ee86693a17ff30 not found: ID does not exist" Jul 23 12:09:06.618437 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.618368 2554 scope.go:117] "RemoveContainer" containerID="93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6" Jul 23 12:09:06.618898 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:09:06.618785 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6\": container with ID starting with 93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6 not found: ID does not exist" containerID="93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6" Jul 23 12:09:06.618898 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.618805 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6"} err="failed to get container status \"93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6\": rpc error: code = NotFound desc = could not find container \"93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6\": container with ID starting with 93db9e21626c362860711e9ecde073ac24f833ffd874663476e99a1fa83775b6 not found: ID does not exist" Jul 23 12:09:06.709856 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709836 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/74580157-9acd-488b-b644-2af27f13f6c4-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.709856 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709856 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709864 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709873 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709880 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/74580157-9acd-488b-b644-2af27f13f6c4-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709888 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709896 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709904 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4smd\" (UniqueName: \"kubernetes.io/projected/74580157-9acd-488b-b644-2af27f13f6c4-kube-api-access-j4smd\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.710013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.709913 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/74580157-9acd-488b-b644-2af27f13f6c4-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:09:06.919739 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.919710 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj"] Jul 23 12:09:06.923828 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:06.923806 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-110e698d-openshift-default-664dbbf8c-wzzhj"] Jul 23 12:09:08.453524 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:08.453489 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74580157-9acd-488b-b644-2af27f13f6c4" path="/var/lib/kubelet/pods/74580157-9acd-488b-b644-2af27f13f6c4/volumes" Jul 23 12:09:22.552285 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552250 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv"] Jul 23 12:09:22.552670 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552657 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" Jul 23 12:09:22.552726 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552672 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eac4e47-7550-4963-9cc5-7350f0b1cb58" containerName="istio-proxy" Jul 23 12:09:22.552726 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552683 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" Jul 23 12:09:22.552726 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552688 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" Jul 23 12:09:22.552726 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552696 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" Jul 23 12:09:22.552726 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552702 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" Jul 23 12:09:22.552883 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552755 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" Jul 23 12:09:22.552883 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.552766 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed089633-3894-4db3-b635-85e28d93cea4" containerName="istio-proxy" Jul 23 12:09:22.555657 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.555614 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.558068 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.558049 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-66b55d3d-openshift-default-dockercfg-dvjc4\"" Jul 23 12:09:22.573732 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.573709 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv"] Jul 23 12:09:22.636267 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.636230 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-envoy\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.636535 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.636513 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-podinfo\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.636848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.636827 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-certs\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.636982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.636969 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-credential-socket\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.637094 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.637082 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istiod-ca-cert\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.637208 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.637195 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9kbs\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-kube-api-access-r9kbs\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.637298 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.637287 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-token\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.637519 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.637502 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-data\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.637664 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.637650 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-socket\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.738716 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738686 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9kbs\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-kube-api-access-r9kbs\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.738869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738723 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-token\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.738869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738762 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-data\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.738869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738786 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-socket\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.738869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738828 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-envoy\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.738869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738849 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-podinfo\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739175 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738917 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-certs\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739175 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738946 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-credential-socket\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739175 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.738999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istiod-ca-cert\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739345 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.739315 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-data\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739401 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.739364 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-certs\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739497 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.739469 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-credential-socket\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.739651 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.739536 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-socket\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.740059 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.740037 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istiod-ca-cert\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.741287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.741256 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-envoy\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.741554 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.741538 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-podinfo\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.753270 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.753246 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-token\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.754245 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.754227 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9kbs\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-kube-api-access-r9kbs\") pod \"e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:22.866062 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:22.865999 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:23.033344 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.033306 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv"] Jul 23 12:09:23.034344 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:09:23.034322 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad17d9b2_4d16_4e25_94b7_928b8881a47e.slice/crio-0ae0d5a173e18f6aea0172958003bb9b3b60644d7883732fa12e434f4aabc7c5 WatchSource:0}: Error finding container 0ae0d5a173e18f6aea0172958003bb9b3b60644d7883732fa12e434f4aabc7c5: Status 404 returned error can't find the container with id 0ae0d5a173e18f6aea0172958003bb9b3b60644d7883732fa12e434f4aabc7c5 Jul 23 12:09:23.036372 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.036342 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:09:23.036453 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.036405 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:09:23.036453 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.036432 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:09:23.683202 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.683166 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" event={"ID":"ad17d9b2-4d16-4e25-94b7-928b8881a47e","Type":"ContainerStarted","Data":"e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f"} Jul 23 12:09:23.683676 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.683215 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" event={"ID":"ad17d9b2-4d16-4e25-94b7-928b8881a47e","Type":"ContainerStarted","Data":"0ae0d5a173e18f6aea0172958003bb9b3b60644d7883732fa12e434f4aabc7c5"} Jul 23 12:09:23.866695 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:23.866664 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:24.871465 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:24.871426 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:24.895996 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:24.895944 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podStartSLOduration=2.895928956 podStartE2EDuration="2.895928956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:09:23.718867538 +0000 UTC m=+2011.824193950" watchObservedRunningTime="2026-07-23 12:09:24.895928956 +0000 UTC m=+2013.001255369" Jul 23 12:09:25.693810 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:25.693778 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:25.694794 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:25.694767 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:38.761234 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:38.761201 2554 generic.go:356] "Generic (PLEG): container finished" podID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerID="e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f" exitCode=0 Jul 23 12:09:38.761641 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:38.761280 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" event={"ID":"ad17d9b2-4d16-4e25-94b7-928b8881a47e","Type":"ContainerDied","Data":"e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f"} Jul 23 12:09:38.761703 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:38.761687 2554 scope.go:117] "RemoveContainer" containerID="e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f" Jul 23 12:09:38.762636 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:38.762598 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:09:38.762706 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:38.762692 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:09:38.762746 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:38.762730 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:09:39.768132 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:39.768101 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" event={"ID":"ad17d9b2-4d16-4e25-94b7-928b8881a47e","Type":"ContainerStarted","Data":"f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985"} Jul 23 12:09:39.867051 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:39.867025 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:39.867904 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:39.867881 2554 patch_prober.go:28] interesting pod/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 23 12:09:39.868031 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:39.867930 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 23 12:09:40.693913 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:40.693828 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:40.867109 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:40.867077 2554 patch_prober.go:28] interesting pod/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 23 12:09:40.867471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:40.867131 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 23 12:09:41.866671 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:41.866611 2554 patch_prober.go:28] interesting pod/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 23 12:09:41.866853 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:41.866687 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 23 12:09:42.866854 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:42.866820 2554 patch_prober.go:28] interesting pod/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 23 12:09:42.867230 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:42.866886 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 23 12:09:43.866484 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:43.866453 2554 patch_prober.go:28] interesting pod/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 23 12:09:43.866701 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:43.866525 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 23 12:09:44.869929 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:44.869900 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:09:44.870885 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:09:44.870854 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:11:51.028043 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:51.027964 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv"] Jul 23 12:11:51.028520 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:51.028218 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" containerID="cri-o://f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985" gracePeriod=30 Jul 23 12:11:56.298648 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.298563 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:11:56.366984 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.366949 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-envoy\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.366984 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.366984 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-podinfo\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367047 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-token\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367071 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-certs\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367112 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-socket\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367143 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-credential-socket\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367171 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istiod-ca-cert\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367202 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-data\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367500 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367235 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9kbs\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-kube-api-access-r9kbs\") pod \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\" (UID: \"ad17d9b2-4d16-4e25-94b7-928b8881a47e\") " Jul 23 12:11:56.367500 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367430 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:56.367500 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367454 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:56.367732 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367661 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.367732 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367683 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.367969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.367884 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:56.368180 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.368147 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:11:56.368459 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.368436 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-data" (OuterVolumeSpecName: "istio-data") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:56.369433 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.369410 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:11:56.369830 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.369806 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-token" (OuterVolumeSpecName: "istio-token") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:56.369923 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.369882 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:11:56.370060 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.370044 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-kube-api-access-r9kbs" (OuterVolumeSpecName: "kube-api-access-r9kbs") pod "ad17d9b2-4d16-4e25-94b7-928b8881a47e" (UID: "ad17d9b2-4d16-4e25-94b7-928b8881a47e"). InnerVolumeSpecName "kube-api-access-r9kbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:11:56.468513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468492 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.468513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468515 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.468672 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468523 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.468672 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468532 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.468672 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468540 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.468672 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468549 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad17d9b2-4d16-4e25-94b7-928b8881a47e-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.468672 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.468556 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9kbs\" (UniqueName: \"kubernetes.io/projected/ad17d9b2-4d16-4e25-94b7-928b8881a47e-kube-api-access-r9kbs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:11:56.503118 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.503079 2554 generic.go:356] "Generic (PLEG): container finished" podID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerID="f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985" exitCode=0 Jul 23 12:11:56.503274 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.503191 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" Jul 23 12:11:56.503274 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.503211 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" event={"ID":"ad17d9b2-4d16-4e25-94b7-928b8881a47e","Type":"ContainerDied","Data":"f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985"} Jul 23 12:11:56.503274 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.503268 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv" event={"ID":"ad17d9b2-4d16-4e25-94b7-928b8881a47e","Type":"ContainerDied","Data":"0ae0d5a173e18f6aea0172958003bb9b3b60644d7883732fa12e434f4aabc7c5"} Jul 23 12:11:56.503435 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.503290 2554 scope.go:117] "RemoveContainer" containerID="f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985" Jul 23 12:11:56.513503 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.513481 2554 scope.go:117] "RemoveContainer" containerID="e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f" Jul 23 12:11:56.521845 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.521825 2554 scope.go:117] "RemoveContainer" containerID="f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985" Jul 23 12:11:56.522121 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:11:56.522098 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985\": container with ID starting with f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985 not found: ID does not exist" containerID="f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985" Jul 23 12:11:56.522177 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.522135 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985"} err="failed to get container status \"f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985\": rpc error: code = NotFound desc = could not find container \"f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985\": container with ID starting with f6d7bd65d553962bec6745868838e47c092e8a8eec49d6655ed13c15c86a6985 not found: ID does not exist" Jul 23 12:11:56.522177 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.522158 2554 scope.go:117] "RemoveContainer" containerID="e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f" Jul 23 12:11:56.522433 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:11:56.522414 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f\": container with ID starting with e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f not found: ID does not exist" containerID="e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f" Jul 23 12:11:56.522534 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.522441 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f"} err="failed to get container status \"e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f\": rpc error: code = NotFound desc = could not find container \"e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f\": container with ID starting with e8589cce164262282c2c6e1db483668abd0dbd8e5046bf7a4ee4080bf1049a9f not found: ID does not exist" Jul 23 12:11:56.527260 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.527237 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv"] Jul 23 12:11:56.532066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:56.532045 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-66b55d3d-openshift-default-64644ddd5f-gkvvv"] Jul 23 12:11:58.454305 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:11:58.454268 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" path="/var/lib/kubelet/pods/ad17d9b2-4d16-4e25-94b7-928b8881a47e/volumes" Jul 23 12:12:47.218712 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.218661 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w"] Jul 23 12:12:47.219320 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.219299 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" Jul 23 12:12:47.219379 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.219323 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" Jul 23 12:12:47.219418 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.219411 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" Jul 23 12:12:47.219457 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.219420 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="74580157-9acd-488b-b644-2af27f13f6c4" containerName="istio-proxy" Jul 23 12:12:47.222826 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.222795 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.230017 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.229982 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-9d4f98-openshift-default-dockercfg-t25hh\"" Jul 23 12:12:47.250218 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.250180 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w"] Jul 23 12:12:47.253762 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253721 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istiod-ca-cert\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.253969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253782 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-envoy\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.253969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253845 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-credential-socket\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.253969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253879 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-certs\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.253969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253905 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2r8c\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-kube-api-access-d2r8c\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.253969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253934 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-podinfo\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.254261 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.253978 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-token\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.254261 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.254027 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-data\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.254261 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.254050 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-socket\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.354853 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.354814 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istiod-ca-cert\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.354863 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-envoy\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.354912 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-credential-socket\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.354935 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-certs\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.354962 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2r8c\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-kube-api-access-d2r8c\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.354988 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-podinfo\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355066 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355039 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-token\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355335 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355095 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-data\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355335 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355131 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-socket\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355734 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355665 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-credential-socket\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355844 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355736 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istiod-ca-cert\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355844 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355786 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-data\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355844 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355797 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-certs\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.355963 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.355839 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-socket\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.358006 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.357967 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-envoy\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.358189 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.358168 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-podinfo\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.365373 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.365335 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2r8c\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-kube-api-access-d2r8c\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.366594 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.366562 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-token\") pod \"e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.534796 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.534748 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:47.698996 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.698811 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w"] Jul 23 12:12:47.701853 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:12:47.701815 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef098a11_e5ba_4f4c_a3a8_0372a5448b46.slice/crio-5636dc456d6bf6b9261d826749db868eaa19e4085168914c2da1eb3e7ce20eb8 WatchSource:0}: Error finding container 5636dc456d6bf6b9261d826749db868eaa19e4085168914c2da1eb3e7ce20eb8: Status 404 returned error can't find the container with id 5636dc456d6bf6b9261d826749db868eaa19e4085168914c2da1eb3e7ce20eb8 Jul 23 12:12:47.704524 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.704478 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:12:47.704668 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.704582 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:12:47.704668 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.704647 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:12:47.786869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:47.786763 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" event={"ID":"ef098a11-e5ba-4f4c-a3a8-0372a5448b46","Type":"ContainerStarted","Data":"5636dc456d6bf6b9261d826749db868eaa19e4085168914c2da1eb3e7ce20eb8"} Jul 23 12:12:48.795013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:48.794972 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" event={"ID":"ef098a11-e5ba-4f4c-a3a8-0372a5448b46","Type":"ContainerStarted","Data":"dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd"} Jul 23 12:12:48.817818 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:48.817762 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podStartSLOduration=1.81774274 podStartE2EDuration="1.81774274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:12:48.816260103 +0000 UTC m=+2216.921586535" watchObservedRunningTime="2026-07-23 12:12:48.81774274 +0000 UTC m=+2216.923069151" Jul 23 12:12:49.535641 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:49.535593 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:49.541282 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:49.541251 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:49.800860 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:49.800772 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:49.801787 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:49.801766 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:12:56.075018 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.074977 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4"] Jul 23 12:12:56.075448 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.075434 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" Jul 23 12:12:56.075448 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.075447 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" Jul 23 12:12:56.075548 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.075536 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad17d9b2-4d16-4e25-94b7-928b8881a47e" containerName="istio-proxy" Jul 23 12:12:56.078650 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.078595 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.088253 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.088119 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-480488-openshift-default-dockercfg-gzwk5\"" Jul 23 12:12:56.100528 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.100483 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4"] Jul 23 12:12:56.145613 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145563 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e209bc9a-4523-495e-8aa1-842bf45e15fb-istiod-ca-cert\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145613 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145618 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-certs\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145673 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-credential-socket\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145695 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-envoy\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145726 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-data\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145775 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-podinfo\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145817 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-token\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145841 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2xkl\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-kube-api-access-g2xkl\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.145905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.145867 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-socket\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246396 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246362 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e209bc9a-4523-495e-8aa1-842bf45e15fb-istiod-ca-cert\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246396 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246406 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-certs\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246677 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-credential-socket\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246677 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246553 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-envoy\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246677 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246650 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-data\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246856 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246684 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-podinfo\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246856 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246720 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-token\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246856 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246757 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2xkl\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-kube-api-access-g2xkl\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.246856 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.246799 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-socket\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.247452 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.247245 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-certs\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.247452 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.247282 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-credential-socket\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.247452 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.247443 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-data\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.247656 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.247498 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-socket\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.247656 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.247617 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e209bc9a-4523-495e-8aa1-842bf45e15fb-istiod-ca-cert\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.249187 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.249157 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-envoy\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.249599 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.249580 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-podinfo\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.256570 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.256533 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2xkl\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-kube-api-access-g2xkl\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.256570 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.256559 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-token\") pod \"e2e-inf-b-480488-openshift-default-d5ff56667-v68r4\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.410544 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.410435 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:56.607198 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.606550 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4"] Jul 23 12:12:56.610480 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:12:56.610405 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode209bc9a_4523_495e_8aa1_842bf45e15fb.slice/crio-92c2850729e377459820a67945c429bdad6c7d6828d136602e51b6c17a920f46 WatchSource:0}: Error finding container 92c2850729e377459820a67945c429bdad6c7d6828d136602e51b6c17a920f46: Status 404 returned error can't find the container with id 92c2850729e377459820a67945c429bdad6c7d6828d136602e51b6c17a920f46 Jul 23 12:12:56.613521 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.613480 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:12:56.613690 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.613579 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:12:56.613690 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.613669 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:12:56.841072 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.841030 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" event={"ID":"e209bc9a-4523-495e-8aa1-842bf45e15fb","Type":"ContainerStarted","Data":"5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362"} Jul 23 12:12:56.841274 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.841102 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" event={"ID":"e209bc9a-4523-495e-8aa1-842bf45e15fb","Type":"ContainerStarted","Data":"92c2850729e377459820a67945c429bdad6c7d6828d136602e51b6c17a920f46"} Jul 23 12:12:56.863216 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:56.863138 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podStartSLOduration=0.863111138 podStartE2EDuration="863.111138ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:12:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:12:56.860536938 +0000 UTC m=+2224.965863388" watchObservedRunningTime="2026-07-23 12:12:56.863111138 +0000 UTC m=+2224.968437550" Jul 23 12:12:57.410835 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:57.410790 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:57.416837 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:57.416804 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:57.847335 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:57.847298 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:12:57.848567 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:12:57.848545 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:13:03.888255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:03.886956 2554 generic.go:356] "Generic (PLEG): container finished" podID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerID="dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd" exitCode=0 Jul 23 12:13:03.888255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:03.887032 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" event={"ID":"ef098a11-e5ba-4f4c-a3a8-0372a5448b46","Type":"ContainerDied","Data":"dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd"} Jul 23 12:13:03.888255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:03.887555 2554 scope.go:117] "RemoveContainer" containerID="dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd" Jul 23 12:13:03.895887 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:03.890820 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:13:03.895887 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:03.890961 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:13:03.895887 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:03.891014 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:13:04.801647 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:04.801586 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:13:04.899725 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:04.899677 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" event={"ID":"ef098a11-e5ba-4f4c-a3a8-0372a5448b46","Type":"ContainerStarted","Data":"223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830"} Jul 23 12:13:05.535788 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:05.535731 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:13:05.536943 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:05.536913 2554 patch_prober.go:28] interesting pod/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 23 12:13:05.537067 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:05.536974 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 23 12:13:06.535939 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:06.535894 2554 patch_prober.go:28] interesting pod/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 23 12:13:06.536347 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:06.535966 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 23 12:13:07.536157 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:07.536110 2554 patch_prober.go:28] interesting pod/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 23 12:13:07.536569 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:07.536170 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 23 12:13:08.535780 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:08.535740 2554 patch_prober.go:28] interesting pod/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 23 12:13:08.535983 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:08.535812 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 23 12:13:09.540517 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:09.540486 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:13:09.541045 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:09.540883 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:13:09.541918 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:09.541895 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:13:11.945713 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:11.945655 2554 generic.go:356] "Generic (PLEG): container finished" podID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerID="5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362" exitCode=0 Jul 23 12:13:11.946265 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:11.945725 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" event={"ID":"e209bc9a-4523-495e-8aa1-842bf45e15fb","Type":"ContainerDied","Data":"5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362"} Jul 23 12:13:11.946265 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:11.946172 2554 scope.go:117] "RemoveContainer" containerID="5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362" Jul 23 12:13:11.947347 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:11.947315 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:13:11.947419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:11.947390 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:13:11.947479 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:11.947421 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:13:12.847919 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:12.847883 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:13:12.953688 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:12.953644 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" event={"ID":"e209bc9a-4523-495e-8aa1-842bf45e15fb","Type":"ContainerStarted","Data":"d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81"} Jul 23 12:13:13.410685 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:13.410613 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:13:13.412223 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:13.412183 2554 patch_prober.go:28] interesting pod/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 23 12:13:13.412398 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:13.412271 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 23 12:13:14.411407 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:14.411366 2554 patch_prober.go:28] interesting pod/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 23 12:13:14.412046 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:14.412012 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 23 12:13:15.411875 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:15.411831 2554 patch_prober.go:28] interesting pod/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 23 12:13:15.412309 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:15.411934 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 23 12:13:16.411666 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:16.411239 2554 patch_prober.go:28] interesting pod/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 23 12:13:16.411666 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:16.411299 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 23 12:13:17.415944 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:17.415909 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:13:17.416424 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:17.416205 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:13:17.417292 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:13:17.417268 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:14:37.667166 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:37.667125 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w"] Jul 23 12:14:37.667728 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:37.667415 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" containerID="cri-o://223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830" gracePeriod=30 Jul 23 12:14:39.543513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:39.543470 2554 patch_prober.go:28] interesting pod/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:14:39.544014 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:39.543544 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:14:42.944006 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:42.943924 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:14:43.007517 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007487 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-token\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.007747 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007533 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-certs\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.007747 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007584 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istiod-ca-cert\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.007747 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007612 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-envoy\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.007747 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007654 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-credential-socket\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.007747 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007716 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-data\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.008026 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007753 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-socket\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.008026 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007800 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-podinfo\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.008026 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.007825 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d2r8c\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-kube-api-access-d2r8c\") pod \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\" (UID: \"ef098a11-e5ba-4f4c-a3a8-0372a5448b46\") " Jul 23 12:14:43.008220 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.008191 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:43.008325 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.008290 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:14:43.008603 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.008546 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:43.009713 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.009527 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-data" (OuterVolumeSpecName: "istio-data") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:43.009840 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.009728 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:43.010741 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.010710 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:14:43.010866 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.010828 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-token" (OuterVolumeSpecName: "istio-token") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:14:43.010866 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.010844 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-kube-api-access-d2r8c" (OuterVolumeSpecName: "kube-api-access-d2r8c") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "kube-api-access-d2r8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:14:43.011076 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.011044 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ef098a11-e5ba-4f4c-a3a8-0372a5448b46" (UID: "ef098a11-e5ba-4f4c-a3a8-0372a5448b46"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:14:43.109199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109157 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109187 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109197 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109206 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2r8c\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-kube-api-access-d2r8c\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109547 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109216 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109547 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109224 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109547 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109231 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109547 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109238 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.109547 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.109246 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef098a11-e5ba-4f4c-a3a8-0372a5448b46-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:14:43.461342 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.461303 2554 generic.go:356] "Generic (PLEG): container finished" podID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerID="223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830" exitCode=0 Jul 23 12:14:43.461554 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.461358 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" Jul 23 12:14:43.461554 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.461390 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" event={"ID":"ef098a11-e5ba-4f4c-a3a8-0372a5448b46","Type":"ContainerDied","Data":"223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830"} Jul 23 12:14:43.461554 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.461457 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w" event={"ID":"ef098a11-e5ba-4f4c-a3a8-0372a5448b46","Type":"ContainerDied","Data":"5636dc456d6bf6b9261d826749db868eaa19e4085168914c2da1eb3e7ce20eb8"} Jul 23 12:14:43.461554 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.461482 2554 scope.go:117] "RemoveContainer" containerID="223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830" Jul 23 12:14:43.472115 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.472086 2554 scope.go:117] "RemoveContainer" containerID="dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd" Jul 23 12:14:43.488196 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.488070 2554 scope.go:117] "RemoveContainer" containerID="223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830" Jul 23 12:14:43.488791 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.488758 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w"] Jul 23 12:14:43.489051 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:14:43.488807 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830\": container with ID starting with 223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830 not found: ID does not exist" containerID="223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830" Jul 23 12:14:43.489136 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.489071 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830"} err="failed to get container status \"223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830\": rpc error: code = NotFound desc = could not find container \"223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830\": container with ID starting with 223e48cda7b0b80bfbd63b44a6c4e61cb7bb9294248633522b39f2149a89b830 not found: ID does not exist" Jul 23 12:14:43.489136 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.489098 2554 scope.go:117] "RemoveContainer" containerID="dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd" Jul 23 12:14:43.490032 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:14:43.490002 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd\": container with ID starting with dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd not found: ID does not exist" containerID="dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd" Jul 23 12:14:43.490193 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.490170 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd"} err="failed to get container status \"dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd\": rpc error: code = NotFound desc = could not find container \"dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd\": container with ID starting with dcefd0b138db56c6ffe8d2af414babaa81e7a0dff7296976588f8f178e3ea9bd not found: ID does not exist" Jul 23 12:14:43.492857 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:43.492825 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-9d4f98-openshift-default-58c69bbb85-5zn5w"] Jul 23 12:14:44.455368 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:14:44.455317 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" path="/var/lib/kubelet/pods/ef098a11-e5ba-4f4c-a3a8-0372a5448b46/volumes" Jul 23 12:15:00.091799 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:00.091750 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4"] Jul 23 12:15:00.092248 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:00.092029 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" containerID="cri-o://d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81" gracePeriod=30 Jul 23 12:15:01.063059 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063009 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm"] Jul 23 12:15:01.063534 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063515 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" Jul 23 12:15:01.063678 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063537 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" Jul 23 12:15:01.063678 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063551 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" Jul 23 12:15:01.063678 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063559 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" Jul 23 12:15:01.063846 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063741 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" Jul 23 12:15:01.063846 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.063759 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef098a11-e5ba-4f4c-a3a8-0372a5448b46" containerName="istio-proxy" Jul 23 12:15:01.067016 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.066989 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.069549 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.069522 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-091828-openshift-default-dockercfg-hlfqt\"" Jul 23 12:15:01.085099 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.085057 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm"] Jul 23 12:15:01.184272 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.184218 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-credential-socket\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.184934 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.184876 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-envoy\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.185182 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.185164 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istiod-ca-cert\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.185381 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.185338 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-data\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.185588 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.185574 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-podinfo\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.185819 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.185799 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djj2q\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-kube-api-access-djj2q\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.186002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.185968 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-certs\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.186159 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.186145 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-socket\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.186349 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.186333 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-token\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.287686 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287642 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djj2q\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-kube-api-access-djj2q\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.287864 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287710 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-certs\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.287864 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287739 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-socket\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.287864 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287773 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-token\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288011 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287925 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-credential-socket\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288011 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287970 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-envoy\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288011 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.287999 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istiod-ca-cert\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288125 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-data\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288125 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288119 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-podinfo\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288215 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288200 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-certs\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288273 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288249 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-socket\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288292 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-credential-socket\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288479 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288458 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-data\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.288728 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.288710 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istiod-ca-cert\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.290584 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.290553 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-envoy\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.290737 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.290718 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-podinfo\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.296395 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.296367 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-token\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.296546 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.296444 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-djj2q\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-kube-api-access-djj2q\") pod \"e2e-api-a-091828-openshift-default-5fb479648c-d29bm\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.378585 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.378485 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:01.539692 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.539661 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm"] Jul 23 12:15:01.542392 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:15:01.542355 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd22a2fd_1c4d_44f9_b8a1_8de0939b3e3b.slice/crio-1596dafc7254acb32b24658d9193b8c1680648e49ca3bd84d15daecca2f73bad WatchSource:0}: Error finding container 1596dafc7254acb32b24658d9193b8c1680648e49ca3bd84d15daecca2f73bad: Status 404 returned error can't find the container with id 1596dafc7254acb32b24658d9193b8c1680648e49ca3bd84d15daecca2f73bad Jul 23 12:15:01.545234 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.545194 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:01.545355 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.545272 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:01.545355 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.545303 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:01.560757 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:01.560722 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" event={"ID":"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b","Type":"ContainerStarted","Data":"1596dafc7254acb32b24658d9193b8c1680648e49ca3bd84d15daecca2f73bad"} Jul 23 12:15:02.419244 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:02.419199 2554 patch_prober.go:28] interesting pod/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:15:02.419660 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:02.419264 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:15:02.568033 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:02.567996 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" event={"ID":"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b","Type":"ContainerStarted","Data":"a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f"} Jul 23 12:15:02.594335 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:02.594274 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podStartSLOduration=1.594253095 podStartE2EDuration="1.594253095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:15:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:15:02.592804676 +0000 UTC m=+2350.698131132" watchObservedRunningTime="2026-07-23 12:15:02.594253095 +0000 UTC m=+2350.699579506" Jul 23 12:15:03.379619 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:03.379569 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:03.385955 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:03.385918 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:03.573919 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:03.573887 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:03.575073 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:03.575052 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:05.361077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.361000 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:15:05.431167 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431131 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-podinfo\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431341 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431201 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-envoy\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431341 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431305 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e209bc9a-4523-495e-8aa1-842bf45e15fb-istiod-ca-cert\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431343 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-socket\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431384 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2xkl\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-kube-api-access-g2xkl\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431425 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-token\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431449 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-data\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431477 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-certs\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431809 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431509 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-credential-socket\") pod \"e209bc9a-4523-495e-8aa1-842bf45e15fb\" (UID: \"e209bc9a-4523-495e-8aa1-842bf45e15fb\") " Jul 23 12:15:05.431871 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431812 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:05.431871 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431832 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e209bc9a-4523-495e-8aa1-842bf45e15fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:15:05.431973 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431928 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e209bc9a-4523-495e-8aa1-842bf45e15fb-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.431973 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.431950 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.432211 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.432044 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:05.432288 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.432186 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:05.432645 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.432594 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:05.433728 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.433690 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-kube-api-access-g2xkl" (OuterVolumeSpecName: "kube-api-access-g2xkl") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "kube-api-access-g2xkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:15:05.433728 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.433704 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:05.434103 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.434077 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:15:05.434207 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.434086 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "e209bc9a-4523-495e-8aa1-842bf45e15fb" (UID: "e209bc9a-4523-495e-8aa1-842bf45e15fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:15:05.532602 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532563 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.532602 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532593 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.532602 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532603 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2xkl\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-kube-api-access-g2xkl\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.532873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532617 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.532873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532667 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.532873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532676 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.532873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.532684 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e209bc9a-4523-495e-8aa1-842bf45e15fb-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:05.586431 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.586397 2554 generic.go:356] "Generic (PLEG): container finished" podID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerID="d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81" exitCode=0 Jul 23 12:15:05.586650 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.586451 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" Jul 23 12:15:05.586650 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.586488 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" event={"ID":"e209bc9a-4523-495e-8aa1-842bf45e15fb","Type":"ContainerDied","Data":"d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81"} Jul 23 12:15:05.586650 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.586548 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4" event={"ID":"e209bc9a-4523-495e-8aa1-842bf45e15fb","Type":"ContainerDied","Data":"92c2850729e377459820a67945c429bdad6c7d6828d136602e51b6c17a920f46"} Jul 23 12:15:05.586650 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.586580 2554 scope.go:117] "RemoveContainer" containerID="d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81" Jul 23 12:15:05.596926 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.596906 2554 scope.go:117] "RemoveContainer" containerID="5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362" Jul 23 12:15:05.606847 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.606816 2554 scope.go:117] "RemoveContainer" containerID="d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81" Jul 23 12:15:05.607237 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:15:05.607213 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81\": container with ID starting with d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81 not found: ID does not exist" containerID="d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81" Jul 23 12:15:05.607289 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.607248 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81"} err="failed to get container status \"d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81\": rpc error: code = NotFound desc = could not find container \"d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81\": container with ID starting with d104ddccabbb1d7480b66d68ebe49b6cec2f3e989a676b7adb25b7fc5f99fb81 not found: ID does not exist" Jul 23 12:15:05.607289 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.607267 2554 scope.go:117] "RemoveContainer" containerID="5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362" Jul 23 12:15:05.607525 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:15:05.607504 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362\": container with ID starting with 5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362 not found: ID does not exist" containerID="5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362" Jul 23 12:15:05.607571 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.607532 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362"} err="failed to get container status \"5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362\": rpc error: code = NotFound desc = could not find container \"5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362\": container with ID starting with 5abe21f3303fbc3bcfec36a416c0891b9583c8534dac9353e0dbad3a1eee6362 not found: ID does not exist" Jul 23 12:15:05.621038 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.620954 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4"] Jul 23 12:15:05.631590 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:05.631558 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-480488-openshift-default-d5ff56667-v68r4"] Jul 23 12:15:06.454563 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:06.454520 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" path="/var/lib/kubelet/pods/e209bc9a-4523-495e-8aa1-842bf45e15fb/volumes" Jul 23 12:15:10.303564 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.303528 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75"] Jul 23 12:15:10.304083 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.303974 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" Jul 23 12:15:10.304083 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.303986 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" Jul 23 12:15:10.304083 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.303997 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" Jul 23 12:15:10.304083 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.304003 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" Jul 23 12:15:10.304083 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.304083 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" Jul 23 12:15:10.304253 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.304094 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="e209bc9a-4523-495e-8aa1-842bf45e15fb" containerName="istio-proxy" Jul 23 12:15:10.308376 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.308353 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.314504 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.314470 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c21064-openshift-default-dockercfg-p4nzl\"" Jul 23 12:15:10.412412 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.412372 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75"] Jul 23 12:15:10.481128 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481087 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-credential-socket\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481337 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481204 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-envoy\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481337 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481239 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-istio-token\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481337 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481268 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39605209-fb3b-4949-9796-56d67ec65421-istiod-ca-cert\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481337 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481298 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vl42l\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-kube-api-access-vl42l\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481531 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481385 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-data\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481531 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481414 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-socket\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481531 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481467 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-certs\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.481531 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.481517 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39605209-fb3b-4949-9796-56d67ec65421-istio-podinfo\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583137 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583035 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-socket\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583137 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583089 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-certs\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583137 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583111 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39605209-fb3b-4949-9796-56d67ec65421-istio-podinfo\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583168 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-credential-socket\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583245 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-envoy\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583292 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-istio-token\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583314 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39605209-fb3b-4949-9796-56d67ec65421-istiod-ca-cert\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583347 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vl42l\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-kube-api-access-vl42l\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583431 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-data\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583818 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583524 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-socket\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583818 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583560 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-certs\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.583818 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.583776 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-data\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.584107 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.584068 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-credential-socket\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.584464 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.584441 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39605209-fb3b-4949-9796-56d67ec65421-istiod-ca-cert\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.586507 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.586482 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-envoy\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.587013 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.586992 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39605209-fb3b-4949-9796-56d67ec65421-istio-podinfo\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.600102 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.600071 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vl42l\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-kube-api-access-vl42l\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.600723 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.600705 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-istio-token\") pod \"e2e-api-b-c21064-openshift-default-8599c87f45-7gj75\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.620271 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.620233 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:10.826322 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.826236 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75"] Jul 23 12:15:10.828779 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:15:10.828749 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39605209_fb3b_4949_9796_56d67ec65421.slice/crio-b2b58f2355b98b7b06fd0424e2693065685e8cb3e1533583afe1b4f860607faf WatchSource:0}: Error finding container b2b58f2355b98b7b06fd0424e2693065685e8cb3e1533583afe1b4f860607faf: Status 404 returned error can't find the container with id b2b58f2355b98b7b06fd0424e2693065685e8cb3e1533583afe1b4f860607faf Jul 23 12:15:10.830979 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.830947 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:10.831054 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.831015 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:10.831054 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:10.831045 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:11.623097 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:11.623045 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" event={"ID":"39605209-fb3b-4949-9796-56d67ec65421","Type":"ContainerStarted","Data":"36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd"} Jul 23 12:15:11.623528 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:11.623152 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" event={"ID":"39605209-fb3b-4949-9796-56d67ec65421","Type":"ContainerStarted","Data":"b2b58f2355b98b7b06fd0424e2693065685e8cb3e1533583afe1b4f860607faf"} Jul 23 12:15:11.655469 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:11.655401 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podStartSLOduration=1.655382773 podStartE2EDuration="1.655382773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:15:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:15:11.654425644 +0000 UTC m=+2359.759752055" watchObservedRunningTime="2026-07-23 12:15:11.655382773 +0000 UTC m=+2359.760709185" Jul 23 12:15:12.621256 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:12.621210 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:12.627459 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:12.627425 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:12.630061 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:12.630027 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:12.631108 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:12.631086 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:19.615866 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:19.615820 2554 patch_prober.go:28] interesting pod/e2e-api-a-091828-openshift-default-5fb479648c-d29bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:15:19.616289 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:19.615907 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:15:21.687766 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:21.687731 2554 generic.go:356] "Generic (PLEG): container finished" podID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerID="a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f" exitCode=0 Jul 23 12:15:21.688225 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:21.687810 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" event={"ID":"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b","Type":"ContainerDied","Data":"a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f"} Jul 23 12:15:21.688299 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:21.688248 2554 scope.go:117] "RemoveContainer" containerID="a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f" Jul 23 12:15:21.689354 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:21.689328 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:21.689410 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:21.689399 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:21.689459 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:21.689448 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:22.699502 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:22.699449 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" event={"ID":"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b","Type":"ContainerStarted","Data":"98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78"} Jul 23 12:15:23.378963 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:23.378926 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:23.379377 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:23.379349 2554 patch_prober.go:28] interesting pod/e2e-api-a-091828-openshift-default-5fb479648c-d29bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 23 12:15:23.379595 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:23.379568 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 23 12:15:24.380041 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:24.379998 2554 patch_prober.go:28] interesting pod/e2e-api-a-091828-openshift-default-5fb479648c-d29bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 23 12:15:24.380692 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:24.380656 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 23 12:15:25.379957 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:25.379905 2554 patch_prober.go:28] interesting pod/e2e-api-a-091828-openshift-default-5fb479648c-d29bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 23 12:15:25.380195 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:25.379978 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 23 12:15:26.379510 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:26.379460 2554 patch_prober.go:28] interesting pod/e2e-api-a-091828-openshift-default-5fb479648c-d29bm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 23 12:15:26.379913 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:26.379880 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 23 12:15:27.382992 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:27.382957 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:27.383472 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:27.383450 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:27.384199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:27.384179 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:28.671892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:28.671844 2554 patch_prober.go:28] interesting pod/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:15:28.672409 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:28.671944 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:15:30.753667 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:30.753614 2554 generic.go:356] "Generic (PLEG): container finished" podID="39605209-fb3b-4949-9796-56d67ec65421" containerID="36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd" exitCode=0 Jul 23 12:15:30.753667 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:30.753657 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" event={"ID":"39605209-fb3b-4949-9796-56d67ec65421","Type":"ContainerDied","Data":"36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd"} Jul 23 12:15:30.754145 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:30.754059 2554 scope.go:117] "RemoveContainer" containerID="36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd" Jul 23 12:15:30.755091 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:30.755066 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:30.755169 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:30.755145 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:30.755210 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:30.755187 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:15:31.760349 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:31.760315 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" event={"ID":"39605209-fb3b-4949-9796-56d67ec65421","Type":"ContainerStarted","Data":"8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de"} Jul 23 12:15:32.621383 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:32.621336 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:32.621614 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:32.621594 2554 patch_prober.go:28] interesting pod/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 23 12:15:32.621711 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:32.621675 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 23 12:15:33.621449 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:33.621409 2554 patch_prober.go:28] interesting pod/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 23 12:15:33.621905 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:33.621472 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 23 12:15:34.621581 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:34.621542 2554 patch_prober.go:28] interesting pod/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 23 12:15:34.622016 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:34.621658 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 23 12:15:35.621616 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:35.621580 2554 patch_prober.go:28] interesting pod/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Jul 23 12:15:35.622063 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:35.621676 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Jul 23 12:15:36.624890 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:36.624856 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:36.625269 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:36.625244 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:36.626060 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:36.626039 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:15:53.474424 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:53.474372 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm"] Jul 23 12:15:53.474902 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:53.474706 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" containerID="cri-o://98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78" gracePeriod=30 Jul 23 12:15:57.386339 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:57.386286 2554 patch_prober.go:28] interesting pod/e2e-api-a-091828-openshift-default-5fb479648c-d29bm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:15:57.386777 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:57.386388 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:15:58.747188 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.747114 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:58.791820 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791783 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-data\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791842 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istiod-ca-cert\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791881 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-envoy\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791900 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-token\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791917 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-credential-socket\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791939 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-socket\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792002 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.791999 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-certs\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792303 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792029 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-podinfo\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792303 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792053 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-djj2q\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-kube-api-access-djj2q\") pod \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\" (UID: \"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b\") " Jul 23 12:15:58.792303 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792243 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:58.792303 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792259 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:58.792525 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792358 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:15:58.792525 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792397 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.792525 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792415 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.792525 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792480 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:58.792927 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.792813 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-data" (OuterVolumeSpecName: "istio-data") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:58.794347 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.794310 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-token" (OuterVolumeSpecName: "istio-token") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:15:58.794347 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.794328 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:15:58.794519 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.794450 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:15:58.794671 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.794653 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-kube-api-access-djj2q" (OuterVolumeSpecName: "kube-api-access-djj2q") pod "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" (UID: "fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b"). InnerVolumeSpecName "kube-api-access-djj2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:15:58.893873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893830 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.893873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893866 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.893873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893874 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.893873 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893882 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.894159 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893891 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.894159 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893898 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djj2q\" (UniqueName: \"kubernetes.io/projected/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-kube-api-access-djj2q\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.894159 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.893907 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:15:58.914376 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.914337 2554 generic.go:356] "Generic (PLEG): container finished" podID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerID="98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78" exitCode=0 Jul 23 12:15:58.914576 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.914386 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" Jul 23 12:15:58.914576 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.914429 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" event={"ID":"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b","Type":"ContainerDied","Data":"98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78"} Jul 23 12:15:58.914576 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.914487 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm" event={"ID":"fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b","Type":"ContainerDied","Data":"1596dafc7254acb32b24658d9193b8c1680648e49ca3bd84d15daecca2f73bad"} Jul 23 12:15:58.914576 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.914510 2554 scope.go:117] "RemoveContainer" containerID="98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78" Jul 23 12:15:58.925047 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.925019 2554 scope.go:117] "RemoveContainer" containerID="a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f" Jul 23 12:15:58.934611 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.934585 2554 scope.go:117] "RemoveContainer" containerID="98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78" Jul 23 12:15:58.934994 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:15:58.934944 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78\": container with ID starting with 98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78 not found: ID does not exist" containerID="98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78" Jul 23 12:15:58.935115 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.934985 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78"} err="failed to get container status \"98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78\": rpc error: code = NotFound desc = could not find container \"98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78\": container with ID starting with 98bc5f4449d5fc90b5dd1339dfd81150de58c02370099724fe60bee8953ebc78 not found: ID does not exist" Jul 23 12:15:58.935115 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.935009 2554 scope.go:117] "RemoveContainer" containerID="a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f" Jul 23 12:15:58.935745 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:15:58.935506 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f\": container with ID starting with a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f not found: ID does not exist" containerID="a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f" Jul 23 12:15:58.935745 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.935546 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f"} err="failed to get container status \"a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f\": rpc error: code = NotFound desc = could not find container \"a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f\": container with ID starting with a44a1115c4bff181fafa248664e8a1fa4802c9e7d47c06fb8d6b513fcbef781f not found: ID does not exist" Jul 23 12:15:58.937751 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.937723 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm"] Jul 23 12:15:58.942829 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:15:58.942782 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-091828-openshift-default-5fb479648c-d29bm"] Jul 23 12:16:00.454405 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:00.454360 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" path="/var/lib/kubelet/pods/fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b/volumes" Jul 23 12:16:20.635982 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:20.635886 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75"] Jul 23 12:16:20.636573 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:20.636153 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" containerID="cri-o://8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de" gracePeriod=30 Jul 23 12:16:21.597091 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597044 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp"] Jul 23 12:16:21.597526 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597503 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" Jul 23 12:16:21.597526 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597525 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" Jul 23 12:16:21.597742 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597538 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" Jul 23 12:16:21.597742 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597546 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" Jul 23 12:16:21.597742 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597702 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" Jul 23 12:16:21.597742 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.597716 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd22a2fd-1c4d-44f9-b8a1-8de0939b3e3b" containerName="istio-proxy" Jul 23 12:16:21.602780 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.602747 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.605289 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.605263 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-290648-openshift-default-dockercfg-5bbk7\"" Jul 23 12:16:21.614682 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.614651 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp"] Jul 23 12:16:21.627342 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.627284 2554 patch_prober.go:28] interesting pod/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:16:21.627543 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.627386 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:16:21.720892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.720847 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-token\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.720892 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.720894 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-envoy\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.720916 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-certs\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.721008 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-podinfo\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.721040 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vxn\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-kube-api-access-f2vxn\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.721065 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-data\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.721112 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istiod-ca-cert\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.721186 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-socket\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.721331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.721214 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-credential-socket\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822167 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822119 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-socket\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822167 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822165 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-credential-socket\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822202 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-token\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822234 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-envoy\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822261 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-certs\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822317 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-podinfo\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822339 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vxn\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-kube-api-access-f2vxn\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822719 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822498 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-data\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822719 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822568 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istiod-ca-cert\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822719 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822670 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-credential-socket\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822719 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822694 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-socket\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.822969 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.822948 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-certs\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.823063 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.823040 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-data\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.823323 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.823306 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istiod-ca-cert\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.825071 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.825045 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-envoy\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.825314 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.825288 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-podinfo\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.831289 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.831253 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-token\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.831444 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.831315 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vxn\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-kube-api-access-f2vxn\") pod \"e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:21.917552 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:21.917456 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:22.092213 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:22.092163 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp"] Jul 23 12:16:22.093758 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:16:22.093726 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eb6d6c_e0e1_4d98_8471_4b5cd5f7245c.slice/crio-9bc4067185339d4753b29687f896156bde319a2e2725bf70ef20564ac7b56ea2 WatchSource:0}: Error finding container 9bc4067185339d4753b29687f896156bde319a2e2725bf70ef20564ac7b56ea2: Status 404 returned error can't find the container with id 9bc4067185339d4753b29687f896156bde319a2e2725bf70ef20564ac7b56ea2 Jul 23 12:16:22.096251 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:22.096214 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:22.096350 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:22.096296 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:22.096350 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:22.096325 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:23.048695 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:23.048654 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" event={"ID":"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c","Type":"ContainerStarted","Data":"6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b"} Jul 23 12:16:23.049099 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:23.048724 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" event={"ID":"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c","Type":"ContainerStarted","Data":"9bc4067185339d4753b29687f896156bde319a2e2725bf70ef20564ac7b56ea2"} Jul 23 12:16:23.075246 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:23.075162 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podStartSLOduration=2.07514441 podStartE2EDuration="2.07514441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:16:23.072809517 +0000 UTC m=+2431.178135945" watchObservedRunningTime="2026-07-23 12:16:23.07514441 +0000 UTC m=+2431.180470887" Jul 23 12:16:23.918020 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:23.917982 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:23.923665 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:23.923614 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:24.054273 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:24.054243 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:24.055423 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:24.055402 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:25.929221 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:25.929188 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:16:26.065525 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.065498 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-envoy\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.065848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.065560 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-socket\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.065848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.065602 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-data\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.065848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.065658 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-certs\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.065848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.065706 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39605209-fb3b-4949-9796-56d67ec65421-istiod-ca-cert\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.065848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.065734 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-istio-token\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.066147 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066013 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:16:26.066147 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066128 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39605209-fb3b-4949-9796-56d67ec65421-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:16:26.066258 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066230 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:16:26.066314 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066266 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-credential-socket\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.066401 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066382 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39605209-fb3b-4949-9796-56d67ec65421-istio-podinfo\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.066457 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066422 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vl42l\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-kube-api-access-vl42l\") pod \"39605209-fb3b-4949-9796-56d67ec65421\" (UID: \"39605209-fb3b-4949-9796-56d67ec65421\") " Jul 23 12:16:26.066533 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066506 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-data" (OuterVolumeSpecName: "istio-data") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:16:26.066721 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066672 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:16:26.066878 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066835 2554 generic.go:356] "Generic (PLEG): container finished" podID="39605209-fb3b-4949-9796-56d67ec65421" containerID="8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de" exitCode=0 Jul 23 12:16:26.066953 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066930 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" Jul 23 12:16:26.067021 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.066846 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.067021 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067001 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" event={"ID":"39605209-fb3b-4949-9796-56d67ec65421","Type":"ContainerDied","Data":"8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de"} Jul 23 12:16:26.067287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067056 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75" event={"ID":"39605209-fb3b-4949-9796-56d67ec65421","Type":"ContainerDied","Data":"b2b58f2355b98b7b06fd0424e2693065685e8cb3e1533583afe1b4f860607faf"} Jul 23 12:16:26.067287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067011 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.067287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067104 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.067287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067116 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39605209-fb3b-4949-9796-56d67ec65421-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.067287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067128 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.067287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.067081 2554 scope.go:117] "RemoveContainer" containerID="8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de" Jul 23 12:16:26.068713 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.068686 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:16:26.068834 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.068753 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-istio-token" (OuterVolumeSpecName: "istio-token") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:16:26.069003 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.068974 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39605209-fb3b-4949-9796-56d67ec65421-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:16:26.069283 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.069158 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-kube-api-access-vl42l" (OuterVolumeSpecName: "kube-api-access-vl42l") pod "39605209-fb3b-4949-9796-56d67ec65421" (UID: "39605209-fb3b-4949-9796-56d67ec65421"). InnerVolumeSpecName "kube-api-access-vl42l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:16:26.084908 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.084880 2554 scope.go:117] "RemoveContainer" containerID="36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd" Jul 23 12:16:26.094812 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.094774 2554 scope.go:117] "RemoveContainer" containerID="8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de" Jul 23 12:16:26.095197 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:16:26.095172 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de\": container with ID starting with 8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de not found: ID does not exist" containerID="8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de" Jul 23 12:16:26.095263 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.095207 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de"} err="failed to get container status \"8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de\": rpc error: code = NotFound desc = could not find container \"8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de\": container with ID starting with 8668ce08d2b39bb99e7350599f66666aa9fd93ce5b5b72fbca2cdcb609d418de not found: ID does not exist" Jul 23 12:16:26.095263 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.095227 2554 scope.go:117] "RemoveContainer" containerID="36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd" Jul 23 12:16:26.095528 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:16:26.095503 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd\": container with ID starting with 36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd not found: ID does not exist" containerID="36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd" Jul 23 12:16:26.095583 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.095534 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd"} err="failed to get container status \"36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd\": rpc error: code = NotFound desc = could not find container \"36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd\": container with ID starting with 36d3094e65f6cf35c4b5d70069670fcb50f5e33d43e41cf5c322812b06cab9fd not found: ID does not exist" Jul 23 12:16:26.168506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.168399 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.168506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.168433 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39605209-fb3b-4949-9796-56d67ec65421-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.168506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.168448 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vl42l\" (UniqueName: \"kubernetes.io/projected/39605209-fb3b-4949-9796-56d67ec65421-kube-api-access-vl42l\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.168506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.168462 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39605209-fb3b-4949-9796-56d67ec65421-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:16:26.393246 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.393203 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75"] Jul 23 12:16:26.402973 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.402929 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c21064-openshift-default-8599c87f45-7gj75"] Jul 23 12:16:26.454673 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:26.454528 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39605209-fb3b-4949-9796-56d67ec65421" path="/var/lib/kubelet/pods/39605209-fb3b-4949-9796-56d67ec65421/volumes" Jul 23 12:16:30.291949 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.291909 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478"] Jul 23 12:16:30.292350 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.292336 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" Jul 23 12:16:30.292421 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.292351 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" Jul 23 12:16:30.292458 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.292432 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" Jul 23 12:16:30.296850 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.296824 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.299425 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.299400 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-1e1abd-openshift-default-dockercfg-nkmv2\"" Jul 23 12:16:30.307237 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.307194 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478"] Jul 23 12:16:30.415764 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.415716 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt6qs\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-kube-api-access-gt6qs\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416023 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.415782 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-envoy\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416023 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.415831 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-socket\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416023 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.415901 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51327e9-0642-4055-a0b9-c71324a5297f-istio-podinfo\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416023 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.415974 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-credential-socket\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.416057 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-certs\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.416086 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51327e9-0642-4055-a0b9-c71324a5297f-istiod-ca-cert\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.416117 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-data\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.416255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.416222 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-istio-token\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517269 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517228 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt6qs\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-kube-api-access-gt6qs\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517282 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-envoy\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517320 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-socket\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517352 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51327e9-0642-4055-a0b9-c71324a5297f-istio-podinfo\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517393 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-credential-socket\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517471 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517454 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-certs\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517794 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517483 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51327e9-0642-4055-a0b9-c71324a5297f-istiod-ca-cert\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517794 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517515 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-data\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517794 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517579 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-istio-token\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517950 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517880 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-socket\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.517950 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.517933 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-credential-socket\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.518050 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.518003 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-certs\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.518284 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.518255 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51327e9-0642-4055-a0b9-c71324a5297f-istiod-ca-cert\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.518415 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.518390 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-data\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.520026 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.520003 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-envoy\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.520256 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.520235 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51327e9-0642-4055-a0b9-c71324a5297f-istio-podinfo\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.527185 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.527145 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-istio-token\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.527331 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.527194 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt6qs\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-kube-api-access-gt6qs\") pod \"e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.610945 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.610840 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:30.775463 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.775426 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478"] Jul 23 12:16:30.777470 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:16:30.777441 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb51327e9_0642_4055_a0b9_c71324a5297f.slice/crio-18f6b79c22f7c42c853d536a5973396a9bcd751cdd5ae90bb876654b82fc4f47 WatchSource:0}: Error finding container 18f6b79c22f7c42c853d536a5973396a9bcd751cdd5ae90bb876654b82fc4f47: Status 404 returned error can't find the container with id 18f6b79c22f7c42c853d536a5973396a9bcd751cdd5ae90bb876654b82fc4f47 Jul 23 12:16:30.779958 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.779923 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:30.780079 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.779996 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:30.780079 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:30.780043 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:31.098909 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:31.098866 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" event={"ID":"b51327e9-0642-4055-a0b9-c71324a5297f","Type":"ContainerStarted","Data":"0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45"} Jul 23 12:16:31.099064 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:31.098930 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" event={"ID":"b51327e9-0642-4055-a0b9-c71324a5297f","Type":"ContainerStarted","Data":"18f6b79c22f7c42c853d536a5973396a9bcd751cdd5ae90bb876654b82fc4f47"} Jul 23 12:16:31.119461 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:31.119341 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podStartSLOduration=1.119317527 podStartE2EDuration="1.119317527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:16:31.118367269 +0000 UTC m=+2439.223693703" watchObservedRunningTime="2026-07-23 12:16:31.119317527 +0000 UTC m=+2439.224643939" Jul 23 12:16:31.611594 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:31.611555 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:31.617084 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:31.617049 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:32.104231 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:32.104197 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:32.105385 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:32.105358 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:40.092840 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.092654 2554 patch_prober.go:28] interesting pod/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 23 12:16:40.092840 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.092741 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 23 12:16:40.151014 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.150978 2554 generic.go:356] "Generic (PLEG): container finished" podID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerID="6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b" exitCode=0 Jul 23 12:16:40.151205 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.151032 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" event={"ID":"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c","Type":"ContainerDied","Data":"6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b"} Jul 23 12:16:40.151536 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.151520 2554 scope.go:117] "RemoveContainer" containerID="6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b" Jul 23 12:16:40.152692 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.152663 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:40.152759 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.152744 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:40.152796 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:40.152778 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:41.159590 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:41.159555 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" event={"ID":"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c","Type":"ContainerStarted","Data":"d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea"} Jul 23 12:16:41.918199 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:41.918160 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:41.918661 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:41.918611 2554 patch_prober.go:28] interesting pod/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 23 12:16:41.918882 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:41.918850 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 23 12:16:42.918603 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:42.918552 2554 patch_prober.go:28] interesting pod/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 23 12:16:42.919143 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:42.918646 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 23 12:16:43.918495 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:43.918428 2554 patch_prober.go:28] interesting pod/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 23 12:16:43.918782 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:43.918591 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 23 12:16:44.918744 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:44.918697 2554 patch_prober.go:28] interesting pod/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Jul 23 12:16:44.918985 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:44.918757 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Jul 23 12:16:45.921980 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:45.921944 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:45.922392 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:45.922342 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:45.923375 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:45.923351 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:16:47.106379 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.106285 2554 patch_prober.go:28] interesting pod/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 23 12:16:47.106379 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.106358 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 23 12:16:47.197695 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.197552 2554 generic.go:356] "Generic (PLEG): container finished" podID="b51327e9-0642-4055-a0b9-c71324a5297f" containerID="0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45" exitCode=0 Jul 23 12:16:47.197695 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.197660 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" event={"ID":"b51327e9-0642-4055-a0b9-c71324a5297f","Type":"ContainerDied","Data":"0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45"} Jul 23 12:16:47.198158 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.198133 2554 scope.go:117] "RemoveContainer" containerID="0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45" Jul 23 12:16:47.199302 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.199273 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:47.199387 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.199348 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:47.199387 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:47.199381 2554 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 23 12:16:48.209783 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:48.209744 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" event={"ID":"b51327e9-0642-4055-a0b9-c71324a5297f","Type":"ContainerStarted","Data":"0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21"} Jul 23 12:16:48.611596 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:48.611551 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:48.611851 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:48.611828 2554 patch_prober.go:28] interesting pod/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 23 12:16:48.611909 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:48.611889 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 23 12:16:49.611515 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:49.611476 2554 patch_prober.go:28] interesting pod/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 23 12:16:49.611960 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:49.611533 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 23 12:16:50.611532 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:50.611482 2554 patch_prober.go:28] interesting pod/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 23 12:16:50.611976 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:50.611558 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 23 12:16:51.612307 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:51.612246 2554 patch_prober.go:28] interesting pod/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Jul 23 12:16:51.612988 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:51.612334 2554 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Jul 23 12:16:52.616106 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:52.616070 2554 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:52.616520 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:52.616479 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:16:52.617341 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:16:52.617318 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:23:06.739706 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.739668 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-139800632219056"] Jul 23 12:23:06.740207 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.740094 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" Jul 23 12:23:06.740207 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.740108 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" Jul 23 12:23:06.740207 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.740192 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="39605209-fb3b-4949-9796-56d67ec65421" containerName="istio-proxy" Jul 23 12:23:06.743038 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.743017 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:06.745546 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.745523 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5d7qv\"" Jul 23 12:23:06.748603 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.748573 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632219056"] Jul 23 12:23:06.791128 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.791089 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4w8c\" (UniqueName: \"kubernetes.io/projected/572a8ca4-a024-4850-88eb-1a4b074dba78-kube-api-access-h4w8c\") pod \"test-curl-7832-139800632219056\" (UID: \"572a8ca4-a024-4850-88eb-1a4b074dba78\") " pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:06.892340 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.892300 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w8c\" (UniqueName: \"kubernetes.io/projected/572a8ca4-a024-4850-88eb-1a4b074dba78-kube-api-access-h4w8c\") pod \"test-curl-7832-139800632219056\" (UID: \"572a8ca4-a024-4850-88eb-1a4b074dba78\") " pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:06.902077 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:06.902048 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w8c\" (UniqueName: \"kubernetes.io/projected/572a8ca4-a024-4850-88eb-1a4b074dba78-kube-api-access-h4w8c\") pod \"test-curl-7832-139800632219056\" (UID: \"572a8ca4-a024-4850-88eb-1a4b074dba78\") " pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:07.054255 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:07.054224 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:07.210563 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:07.210534 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632219056"] Jul 23 12:23:07.213287 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:23:07.213256 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod572a8ca4_a024_4850_88eb_1a4b074dba78.slice/crio-4dab2235132f3c5a578256a2796e266d8658cb0e8c4a9ac1262c5ae5ffc8bf1d WatchSource:0}: Error finding container 4dab2235132f3c5a578256a2796e266d8658cb0e8c4a9ac1262c5ae5ffc8bf1d: Status 404 returned error can't find the container with id 4dab2235132f3c5a578256a2796e266d8658cb0e8c4a9ac1262c5ae5ffc8bf1d Jul 23 12:23:07.214996 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:07.214968 2554 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 23 12:23:07.310001 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:07.309884 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632219056" event={"ID":"572a8ca4-a024-4850-88eb-1a4b074dba78","Type":"ContainerStarted","Data":"4dab2235132f3c5a578256a2796e266d8658cb0e8c4a9ac1262c5ae5ffc8bf1d"} Jul 23 12:23:08.316738 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:08.316697 2554 generic.go:356] "Generic (PLEG): container finished" podID="572a8ca4-a024-4850-88eb-1a4b074dba78" containerID="92fe6317bb54a42cb2335260dd885ff9cc4ab579c4b4f95027da0250f96c1c70" exitCode=0 Jul 23 12:23:08.317131 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:08.316784 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632219056" event={"ID":"572a8ca4-a024-4850-88eb-1a4b074dba78","Type":"ContainerDied","Data":"92fe6317bb54a42cb2335260dd885ff9cc4ab579c4b4f95027da0250f96c1c70"} Jul 23 12:23:09.469431 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.469394 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:09.509460 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.509430 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7832-139800632219056_572a8ca4-a024-4850-88eb-1a4b074dba78/test-curl-7832-139800632219056/0.log" Jul 23 12:23:09.518765 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.518730 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h4w8c\" (UniqueName: \"kubernetes.io/projected/572a8ca4-a024-4850-88eb-1a4b074dba78-kube-api-access-h4w8c\") pod \"572a8ca4-a024-4850-88eb-1a4b074dba78\" (UID: \"572a8ca4-a024-4850-88eb-1a4b074dba78\") " Jul 23 12:23:09.521372 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.521321 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572a8ca4-a024-4850-88eb-1a4b074dba78-kube-api-access-h4w8c" (OuterVolumeSpecName: "kube-api-access-h4w8c") pod "572a8ca4-a024-4850-88eb-1a4b074dba78" (UID: "572a8ca4-a024-4850-88eb-1a4b074dba78"). InnerVolumeSpecName "kube-api-access-h4w8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:09.532320 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.532281 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-139800632219056"] Jul 23 12:23:09.536003 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.535967 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-139800632219056"] Jul 23 12:23:09.620361 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.620267 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4w8c\" (UniqueName: \"kubernetes.io/projected/572a8ca4-a024-4850-88eb-1a4b074dba78-kube-api-access-h4w8c\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:09.707513 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.707468 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:09.707946 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.707929 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572a8ca4-a024-4850-88eb-1a4b074dba78" containerName="test-curl-7832-139800632219056" Jul 23 12:23:09.707946 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.707947 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="572a8ca4-a024-4850-88eb-1a4b074dba78" containerName="test-curl-7832-139800632219056" Jul 23 12:23:09.708050 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.708026 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="572a8ca4-a024-4850-88eb-1a4b074dba78" containerName="test-curl-7832-139800632219056" Jul 23 12:23:09.711261 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.711235 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:09.718859 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.718819 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:09.822398 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.822349 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr64d\" (UniqueName: \"kubernetes.io/projected/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00-kube-api-access-vr64d\") pod \"test-curl-7832-139800632017952\" (UID: \"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00\") " pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:09.923182 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.923078 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr64d\" (UniqueName: \"kubernetes.io/projected/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00-kube-api-access-vr64d\") pod \"test-curl-7832-139800632017952\" (UID: \"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00\") " pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:09.932227 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:09.932185 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr64d\" (UniqueName: \"kubernetes.io/projected/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00-kube-api-access-vr64d\") pod \"test-curl-7832-139800632017952\" (UID: \"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00\") " pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:10.023356 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:10.023308 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:10.176651 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:10.176592 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:10.178758 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:23:10.178717 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0cb205e_1ab5_41cc_9b7f_e64ae00c9b00.slice/crio-a5608fb299759664c1f909c0bd3afe3c0fa16cb13eb8ac1453fdcfbd2f8b75d8 WatchSource:0}: Error finding container a5608fb299759664c1f909c0bd3afe3c0fa16cb13eb8ac1453fdcfbd2f8b75d8: Status 404 returned error can't find the container with id a5608fb299759664c1f909c0bd3afe3c0fa16cb13eb8ac1453fdcfbd2f8b75d8 Jul 23 12:23:10.329219 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:10.329183 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:10.329430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:10.329196 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4dab2235132f3c5a578256a2796e266d8658cb0e8c4a9ac1262c5ae5ffc8bf1d" Jul 23 12:23:10.330608 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:10.330573 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632017952" event={"ID":"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00","Type":"ContainerStarted","Data":"a5608fb299759664c1f909c0bd3afe3c0fa16cb13eb8ac1453fdcfbd2f8b75d8"} Jul 23 12:23:10.340782 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:23:10.340742 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-139800632219056\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="572a8ca4-a024-4850-88eb-1a4b074dba78" pod="opendatahub/test-curl-7832-139800632219056" Jul 23 12:23:10.454512 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:10.454430 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572a8ca4-a024-4850-88eb-1a4b074dba78" path="/var/lib/kubelet/pods/572a8ca4-a024-4850-88eb-1a4b074dba78/volumes" Jul 23 12:23:11.337478 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:11.337440 2554 generic.go:356] "Generic (PLEG): container finished" podID="d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" containerID="6f103df8b0dbd13e5d443b5cc2d773b5253c4d50a7e4c7351bb12c31c164f9ec" exitCode=0 Jul 23 12:23:11.338083 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:11.337530 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632017952" event={"ID":"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00","Type":"ContainerDied","Data":"6f103df8b0dbd13e5d443b5cc2d773b5253c4d50a7e4c7351bb12c31c164f9ec"} Jul 23 12:23:12.490080 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.490038 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:12.509298 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.509239 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7832-139800632017952_d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00/test-curl-7832-139800632017952/0.log" Jul 23 12:23:12.531778 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.531736 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:12.535325 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.535288 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:12.548028 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.547986 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vr64d\" (UniqueName: \"kubernetes.io/projected/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00-kube-api-access-vr64d\") pod \"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00\" (UID: \"d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00\") " Jul 23 12:23:12.550364 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.550332 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00-kube-api-access-vr64d" (OuterVolumeSpecName: "kube-api-access-vr64d") pod "d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" (UID: "d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00"). InnerVolumeSpecName "kube-api-access-vr64d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:12.649346 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.649246 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr64d\" (UniqueName: \"kubernetes.io/projected/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00-kube-api-access-vr64d\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:12.662848 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.662799 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:12.663316 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.663299 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" containerName="test-curl-7832-139800632017952" Jul 23 12:23:12.663359 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.663319 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" containerName="test-curl-7832-139800632017952" Jul 23 12:23:12.663419 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.663409 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" containerName="test-curl-7832-139800632017952" Jul 23 12:23:12.666546 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.666523 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:12.670462 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.670429 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:12.750703 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.750658 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdknr\" (UniqueName: \"kubernetes.io/projected/f7ba4f44-0963-4f9b-a366-10e53d7cab87-kube-api-access-fdknr\") pod \"test-curl-7832-139800632017952\" (UID: \"f7ba4f44-0963-4f9b-a366-10e53d7cab87\") " pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:12.852041 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.851995 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdknr\" (UniqueName: \"kubernetes.io/projected/f7ba4f44-0963-4f9b-a366-10e53d7cab87-kube-api-access-fdknr\") pod \"test-curl-7832-139800632017952\" (UID: \"f7ba4f44-0963-4f9b-a366-10e53d7cab87\") " pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:12.860898 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.860857 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdknr\" (UniqueName: \"kubernetes.io/projected/f7ba4f44-0963-4f9b-a366-10e53d7cab87-kube-api-access-fdknr\") pod \"test-curl-7832-139800632017952\" (UID: \"f7ba4f44-0963-4f9b-a366-10e53d7cab87\") " pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:12.978958 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:12.978860 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:13.129082 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:13.129053 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:13.131397 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:23:13.131358 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7ba4f44_0963_4f9b_a366_10e53d7cab87.slice/crio-84a7e69f16948faed1bea54032d9acb669f22f7bddffba20d49d35428c84de06 WatchSource:0}: Error finding container 84a7e69f16948faed1bea54032d9acb669f22f7bddffba20d49d35428c84de06: Status 404 returned error can't find the container with id 84a7e69f16948faed1bea54032d9acb669f22f7bddffba20d49d35428c84de06 Jul 23 12:23:13.349704 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:13.349649 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632017952" event={"ID":"f7ba4f44-0963-4f9b-a366-10e53d7cab87","Type":"ContainerStarted","Data":"84a7e69f16948faed1bea54032d9acb669f22f7bddffba20d49d35428c84de06"} Jul 23 12:23:13.350958 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:13.350932 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5608fb299759664c1f909c0bd3afe3c0fa16cb13eb8ac1453fdcfbd2f8b75d8" Jul 23 12:23:13.350958 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:13.350957 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:13.363814 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:13.363769 2554 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7832-139800632017952" oldPodUID="d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" podUID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" Jul 23 12:23:14.358587 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:14.358553 2554 generic.go:356] "Generic (PLEG): container finished" podID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" containerID="db27f2d3f8a8d1bca2590929b5cc69fdddd9a0c8e59bd493d1758cf28ce33797" exitCode=0 Jul 23 12:23:14.359049 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:14.358614 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7832-139800632017952" event={"ID":"f7ba4f44-0963-4f9b-a366-10e53d7cab87","Type":"ContainerDied","Data":"db27f2d3f8a8d1bca2590929b5cc69fdddd9a0c8e59bd493d1758cf28ce33797"} Jul 23 12:23:14.455040 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:14.454996 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00" path="/var/lib/kubelet/pods/d0cb205e-1ab5-41cc-9b7f-e64ae00c9b00/volumes" Jul 23 12:23:15.519287 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.519252 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:15.542537 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.542506 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7832-139800632017952_f7ba4f44-0963-4f9b-a366-10e53d7cab87/test-curl-7832-139800632017952/0.log" Jul 23 12:23:15.565468 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.565426 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:15.571968 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.571928 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7832-139800632017952"] Jul 23 12:23:15.580911 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.580873 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fdknr\" (UniqueName: \"kubernetes.io/projected/f7ba4f44-0963-4f9b-a366-10e53d7cab87-kube-api-access-fdknr\") pod \"f7ba4f44-0963-4f9b-a366-10e53d7cab87\" (UID: \"f7ba4f44-0963-4f9b-a366-10e53d7cab87\") " Jul 23 12:23:15.583259 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.583229 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7ba4f44-0963-4f9b-a366-10e53d7cab87-kube-api-access-fdknr" (OuterVolumeSpecName: "kube-api-access-fdknr") pod "f7ba4f44-0963-4f9b-a366-10e53d7cab87" (UID: "f7ba4f44-0963-4f9b-a366-10e53d7cab87"). InnerVolumeSpecName "kube-api-access-fdknr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:15.681754 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:15.681613 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdknr\" (UniqueName: \"kubernetes.io/projected/f7ba4f44-0963-4f9b-a366-10e53d7cab87-kube-api-access-fdknr\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:16.371053 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:16.371019 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:16.371053 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:16.371055 2554 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="84a7e69f16948faed1bea54032d9acb669f22f7bddffba20d49d35428c84de06" Jul 23 12:23:16.381016 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:23:16.380975 2554 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7832-139800632017952\" is forbidden: User \"system:node:ip-10-0-141-203.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-203.ec2.internal' and this object" podUID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" pod="opendatahub/test-curl-7832-139800632017952" Jul 23 12:23:16.454381 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:16.454338 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" path="/var/lib/kubelet/pods/f7ba4f44-0963-4f9b-a366-10e53d7cab87/volumes" Jul 23 12:23:43.748145 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:43.748051 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp"] Jul 23 12:23:43.748796 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:43.748326 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" containerID="cri-o://d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea" gracePeriod=30 Jul 23 12:23:45.924609 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:45.924569 2554 patch_prober.go:28] interesting pod/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:23:45.925091 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:45.924703 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:23:49.024213 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.024131 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:23:49.127489 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127446 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-socket\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127713 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127515 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-podinfo\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127713 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127539 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istiod-ca-cert\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127713 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127698 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-certs\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127823 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127754 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-envoy\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127823 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127788 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f2vxn\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-kube-api-access-f2vxn\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127823 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127812 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-data\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127817 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:49.127974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127833 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-credential-socket\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.127974 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.127882 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-token\") pod \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\" (UID: \"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c\") " Jul 23 12:23:49.128133 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128012 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:49.128133 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128022 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:23:49.128310 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128273 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:49.128310 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128284 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.128502 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128328 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.128502 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128346 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.128922 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.128898 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-data" (OuterVolumeSpecName: "istio-data") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:49.130004 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.129980 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:23:49.130163 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.130147 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-token" (OuterVolumeSpecName: "istio-token") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:49.130473 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.130452 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:23:49.130530 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.130462 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-kube-api-access-f2vxn" (OuterVolumeSpecName: "kube-api-access-f2vxn") pod "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" (UID: "02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c"). InnerVolumeSpecName "kube-api-access-f2vxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:23:49.229757 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.229705 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.229757 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.229750 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.229757 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.229761 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vxn\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-kube-api-access-f2vxn\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.229757 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.229770 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.229757 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.229779 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.230087 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.229787 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:23:49.566812 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.566779 2554 generic.go:356] "Generic (PLEG): container finished" podID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerID="d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea" exitCode=0 Jul 23 12:23:49.567010 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.566838 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" Jul 23 12:23:49.567010 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.566838 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" event={"ID":"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c","Type":"ContainerDied","Data":"d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea"} Jul 23 12:23:49.567010 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.566956 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp" event={"ID":"02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c","Type":"ContainerDied","Data":"9bc4067185339d4753b29687f896156bde319a2e2725bf70ef20564ac7b56ea2"} Jul 23 12:23:49.567010 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.566980 2554 scope.go:117] "RemoveContainer" containerID="d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea" Jul 23 12:23:49.577984 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.577963 2554 scope.go:117] "RemoveContainer" containerID="6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b" Jul 23 12:23:49.589572 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.589545 2554 scope.go:117] "RemoveContainer" containerID="d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea" Jul 23 12:23:49.589892 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:23:49.589870 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea\": container with ID starting with d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea not found: ID does not exist" containerID="d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea" Jul 23 12:23:49.589966 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.589902 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea"} err="failed to get container status \"d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea\": rpc error: code = NotFound desc = could not find container \"d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea\": container with ID starting with d8f16d014e5edab5ae24432ee810b7facff5a4e5ae4b2e4b732360046fb273ea not found: ID does not exist" Jul 23 12:23:49.589966 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.589920 2554 scope.go:117] "RemoveContainer" containerID="6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b" Jul 23 12:23:49.590144 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:23:49.590128 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b\": container with ID starting with 6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b not found: ID does not exist" containerID="6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b" Jul 23 12:23:49.590191 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.590157 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b"} err="failed to get container status \"6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b\": rpc error: code = NotFound desc = could not find container \"6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b\": container with ID starting with 6dc49098b3f7792397d35db056e8c38372c1d596c24be0f62fa879c108f3c46b not found: ID does not exist" Jul 23 12:23:49.590517 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.590498 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp"] Jul 23 12:23:49.594798 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:49.594763 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-290648-openshift-default-55f75984bc-kmjqp"] Jul 23 12:23:50.453646 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:23:50.453603 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" path="/var/lib/kubelet/pods/02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c/volumes" Jul 23 12:24:06.333121 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:06.333061 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478"] Jul 23 12:24:06.334035 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:06.333971 2554 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" containerID="cri-o://0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21" gracePeriod=30 Jul 23 12:24:07.020130 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:07.020085 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c4f6c88d-gvthw_4b67f142-0f4c-43ca-830f-1f273a26050a/authorino/0.log" Jul 23 12:24:07.618746 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:07.618704 2554 patch_prober.go:28] interesting pod/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 23 12:24:07.619149 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:07.618775 2554 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 23 12:24:11.599192 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.599109 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:24:11.648236 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648202 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gt6qs\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-kube-api-access-gt6qs\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648236 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648240 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-certs\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648283 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51327e9-0642-4055-a0b9-c71324a5297f-istiod-ca-cert\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648308 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-socket\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648324 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-istio-token\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648351 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-credential-socket\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648375 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51327e9-0642-4055-a0b9-c71324a5297f-istio-podinfo\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648431 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-data\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.648829 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648656 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:24:11.648829 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648673 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:24:11.648829 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648750 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b51327e9-0642-4055-a0b9-c71324a5297f-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 23 12:24:11.648829 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648802 2554 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.648829 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648821 2554 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-workload-certs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.649099 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.648975 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:24:11.649326 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.649302 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-data" (OuterVolumeSpecName: "istio-data") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:24:11.650671 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.650642 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-kube-api-access-gt6qs" (OuterVolumeSpecName: "kube-api-access-gt6qs") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "kube-api-access-gt6qs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:24:11.650823 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.650797 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b51327e9-0642-4055-a0b9-c71324a5297f-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 23 12:24:11.651143 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.651119 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-istio-token" (OuterVolumeSpecName: "istio-token") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 23 12:24:11.685904 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.685868 2554 generic.go:356] "Generic (PLEG): container finished" podID="b51327e9-0642-4055-a0b9-c71324a5297f" containerID="0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21" exitCode=0 Jul 23 12:24:11.686082 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.685936 2554 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" Jul 23 12:24:11.686082 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.685962 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" event={"ID":"b51327e9-0642-4055-a0b9-c71324a5297f","Type":"ContainerDied","Data":"0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21"} Jul 23 12:24:11.686082 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.686055 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478" event={"ID":"b51327e9-0642-4055-a0b9-c71324a5297f","Type":"ContainerDied","Data":"18f6b79c22f7c42c853d536a5973396a9bcd751cdd5ae90bb876654b82fc4f47"} Jul 23 12:24:11.686272 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.686085 2554 scope.go:117] "RemoveContainer" containerID="0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21" Jul 23 12:24:11.696337 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.696309 2554 scope.go:117] "RemoveContainer" containerID="0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45" Jul 23 12:24:11.705505 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.705484 2554 scope.go:117] "RemoveContainer" containerID="0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21" Jul 23 12:24:11.705852 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:24:11.705831 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21\": container with ID starting with 0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21 not found: ID does not exist" containerID="0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21" Jul 23 12:24:11.705907 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.705863 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21"} err="failed to get container status \"0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21\": rpc error: code = NotFound desc = could not find container \"0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21\": container with ID starting with 0ba04b1aa2e65d217db203d306f0f9da1a6f86519b96b0cfe4b94c0dd6025d21 not found: ID does not exist" Jul 23 12:24:11.705907 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.705882 2554 scope.go:117] "RemoveContainer" containerID="0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45" Jul 23 12:24:11.706178 ip-10-0-141-203 kubenswrapper[2554]: E0723 12:24:11.706157 2554 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45\": container with ID starting with 0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45 not found: ID does not exist" containerID="0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45" Jul 23 12:24:11.706219 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.706183 2554 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45"} err="failed to get container status \"0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45\": rpc error: code = NotFound desc = could not find container \"0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45\": container with ID starting with 0656f73b708c3c79fd205212e4c16586c59dca74aecc09a418be0f482246ed45 not found: ID does not exist" Jul 23 12:24:11.749861 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.749822 2554 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-envoy\") pod \"b51327e9-0642-4055-a0b9-c71324a5297f\" (UID: \"b51327e9-0642-4055-a0b9-c71324a5297f\") " Jul 23 12:24:11.750018 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.750008 2554 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gt6qs\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-kube-api-access-gt6qs\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.750061 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.750021 2554 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b51327e9-0642-4055-a0b9-c71324a5297f-istiod-ca-cert\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.750061 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.750030 2554 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b51327e9-0642-4055-a0b9-c71324a5297f-istio-token\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.750061 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.750038 2554 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-credential-socket\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.750061 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.750046 2554 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b51327e9-0642-4055-a0b9-c71324a5297f-istio-podinfo\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.750061 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.750054 2554 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-data\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:11.752070 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.752045 2554 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b51327e9-0642-4055-a0b9-c71324a5297f" (UID: "b51327e9-0642-4055-a0b9-c71324a5297f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 23 12:24:11.851039 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:11.850947 2554 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b51327e9-0642-4055-a0b9-c71324a5297f-istio-envoy\") on node \"ip-10-0-141-203.ec2.internal\" DevicePath \"\"" Jul 23 12:24:12.009130 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:12.009048 2554 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478"] Jul 23 12:24:12.011222 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:12.011189 2554 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-1e1abd-openshift-default-76644d69cf-v7478"] Jul 23 12:24:12.454112 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:12.454075 2554 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" path="/var/lib/kubelet/pods/b51327e9-0642-4055-a0b9-c71324a5297f/volumes" Jul 23 12:24:12.780683 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:12.780605 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t2k79_778eb9a6-8129-4808-948a-470d451f8304/manager/0.log" Jul 23 12:24:12.908544 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:12.908507 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wxvg9_b3834800-cac3-4f1d-8b54-670f8f506e64/manager/0.log" Jul 23 12:24:13.160528 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:13.160501 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dlqj6_28b25b2d-cd54-4ac4-8f63-531946cef94e/server/0.log" Jul 23 12:24:13.288569 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:13.288533 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9cwng_de982565-fe14-4b51-84fa-3ff6144a1617/manager/0.log" Jul 23 12:24:13.408007 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:13.407981 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-n2g77_57c365ab-3761-4a8c-9296-8eda6f086567/copy-manifests/0.log" Jul 23 12:24:13.415168 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:13.415072 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-n2g77_57c365ab-3761-4a8c-9296-8eda6f086567/manager/0.log" Jul 23 12:24:14.428527 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:14.428499 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gxdcc_36651474-2053-4546-a022-c97873821378/postgres/0.log" Jul 23 12:24:15.801004 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:15.800972 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c4f6c88d-gvthw_4b67f142-0f4c-43ca-830f-1f273a26050a/authorino/0.log" Jul 23 12:24:15.925257 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:15.925211 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-cb862_50fe6d81-eab3-4d54-9f7b-65fe0272b4b2/manager/0.log" Jul 23 12:24:16.050685 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:16.050642 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5stpd_444e9960-300a-4ee8-9a5d-464a343316cf/manager/0.log" Jul 23 12:24:16.292571 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:16.292539 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xdwxf_a3ebb31f-0f6c-465d-b304-cff9297e8865/registry-server/0.log" Jul 23 12:24:16.422763 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:16.422714 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ddcdq_4cca4d76-e5f6-466a-ab60-6aae804516c6/manager/0.log" Jul 23 12:24:16.669859 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:16.669768 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6t9fq_9490542d-4d00-4e2e-83a7-294aba9ab51b/manager/0.log" Jul 23 12:24:17.051720 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:17.051667 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4h2ncb_87931ee1-f888-472f-8be2-98c84ac77bd3/istio-proxy/0.log" Jul 23 12:24:17.430307 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:17.430185 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-zxjwr_3cfd86c0-746c-46e9-a026-487b5d3f81ff/kube-auth-proxy/0.log" Jul 23 12:24:17.561657 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:17.561583 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/1.log" Jul 23 12:24:18.558019 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:18.557989 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-8bsjf_047434b9-606d-418f-a7e7-17dfebbcd8d5/main/0.log" Jul 23 12:24:25.321906 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.321865 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jgf7/must-gather-2f79g"] Jul 23 12:24:25.322296 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322260 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" Jul 23 12:24:25.322296 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322272 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" Jul 23 12:24:25.322296 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322282 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" Jul 23 12:24:25.322296 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322288 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322301 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" containerName="test-curl-7832-139800632017952" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322308 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" containerName="test-curl-7832-139800632017952" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322316 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322321 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322386 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322397 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7ba4f44-0963-4f9b-a366-10e53d7cab87" containerName="test-curl-7832-139800632017952" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322402 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322410 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" Jul 23 12:24:25.322430 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322415 2554 memory_manager.go:368] "RemoveStaleState removing state" podUID="02eb6d6c-e0e1-4d98-8471-4b5cd5f7245c" containerName="istio-proxy" Jul 23 12:24:25.322715 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322477 2554 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" Jul 23 12:24:25.322715 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.322482 2554 state_mem.go:107] "Deleted CPUSet assignment" podUID="b51327e9-0642-4055-a0b9-c71324a5297f" containerName="istio-proxy" Jul 23 12:24:25.327222 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.327197 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.329707 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.329675 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jgf7\"/\"openshift-service-ca.crt\"" Jul 23 12:24:25.330547 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.330518 2554 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-5jgf7\"/\"default-dockercfg-st8ws\"" Jul 23 12:24:25.330681 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.330575 2554 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-5jgf7\"/\"kube-root-ca.crt\"" Jul 23 12:24:25.339053 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.339024 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/must-gather-2f79g"] Jul 23 12:24:25.380349 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.380306 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt9kh\" (UniqueName: \"kubernetes.io/projected/04e688ec-7caf-4265-8fec-cfc850e2d161-kube-api-access-rt9kh\") pod \"must-gather-2f79g\" (UID: \"04e688ec-7caf-4265-8fec-cfc850e2d161\") " pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.380542 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.380422 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04e688ec-7caf-4265-8fec-cfc850e2d161-must-gather-output\") pod \"must-gather-2f79g\" (UID: \"04e688ec-7caf-4265-8fec-cfc850e2d161\") " pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.481881 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.481848 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04e688ec-7caf-4265-8fec-cfc850e2d161-must-gather-output\") pod \"must-gather-2f79g\" (UID: \"04e688ec-7caf-4265-8fec-cfc850e2d161\") " pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.482071 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.481922 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt9kh\" (UniqueName: \"kubernetes.io/projected/04e688ec-7caf-4265-8fec-cfc850e2d161-kube-api-access-rt9kh\") pod \"must-gather-2f79g\" (UID: \"04e688ec-7caf-4265-8fec-cfc850e2d161\") " pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.482228 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.482201 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/04e688ec-7caf-4265-8fec-cfc850e2d161-must-gather-output\") pod \"must-gather-2f79g\" (UID: \"04e688ec-7caf-4265-8fec-cfc850e2d161\") " pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.503560 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.503523 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt9kh\" (UniqueName: \"kubernetes.io/projected/04e688ec-7caf-4265-8fec-cfc850e2d161-kube-api-access-rt9kh\") pod \"must-gather-2f79g\" (UID: \"04e688ec-7caf-4265-8fec-cfc850e2d161\") " pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.638477 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.638381 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/must-gather-2f79g" Jul 23 12:24:25.786602 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:25.786572 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/must-gather-2f79g"] Jul 23 12:24:25.789413 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:24:25.789382 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04e688ec_7caf_4265_8fec_cfc850e2d161.slice/crio-40789181800adf224b71575ae646a8902a65fc4633c6e0cfb05710bac35207e7 WatchSource:0}: Error finding container 40789181800adf224b71575ae646a8902a65fc4633c6e0cfb05710bac35207e7: Status 404 returned error can't find the container with id 40789181800adf224b71575ae646a8902a65fc4633c6e0cfb05710bac35207e7 Jul 23 12:24:26.776981 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:26.776924 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/must-gather-2f79g" event={"ID":"04e688ec-7caf-4265-8fec-cfc850e2d161","Type":"ContainerStarted","Data":"e08e8ae233e1c0f00ff126c8c9b133ee178120dcaaa49e08bd58f0a198796183"} Jul 23 12:24:26.777423 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:26.777005 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/must-gather-2f79g" event={"ID":"04e688ec-7caf-4265-8fec-cfc850e2d161","Type":"ContainerStarted","Data":"40789181800adf224b71575ae646a8902a65fc4633c6e0cfb05710bac35207e7"} Jul 23 12:24:27.785919 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:27.785872 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/must-gather-2f79g" event={"ID":"04e688ec-7caf-4265-8fec-cfc850e2d161","Type":"ContainerStarted","Data":"b2b3ce929188108f2e6df6f9e010c59a0ed157fc73ce3e602a71560bcea66a28"} Jul 23 12:24:27.807291 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:27.807215 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jgf7/must-gather-2f79g" podStartSLOduration=1.9240827 podStartE2EDuration="2.807191348s" totalImagesPullingTime="883.108648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-23 12:24:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:24:27.805446164 +0000 UTC m=+2915.910772575" watchObservedRunningTime="2026-07-23 12:24:27.807191348 +0000 UTC m=+2915.912517758" Jul 23 12:24:28.333427 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:28.333386 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8vr4l_b147eb79-42fd-4c49-a7a1-972af611f9db/global-pull-secret-syncer/0.log" Jul 23 12:24:28.475764 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:28.475727 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dt86d_2a0cf057-5bc8-47b9-b9c6-76e11b9c5916/konnectivity-agent/0.log" Jul 23 12:24:28.567095 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:28.567063 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-203.ec2.internal_356e6d0e35ed5731a5b22ac92b6369fe/haproxy/0.log" Jul 23 12:24:34.054837 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:34.054789 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-74c4f6c88d-gvthw_4b67f142-0f4c-43ca-830f-1f273a26050a/authorino/0.log" Jul 23 12:24:34.084278 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:34.084244 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-cb862_50fe6d81-eab3-4d54-9f7b-65fe0272b4b2/manager/0.log" Jul 23 12:24:34.111888 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:34.111853 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-5stpd_444e9960-300a-4ee8-9a5d-464a343316cf/manager/0.log" Jul 23 12:24:34.199305 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:34.199275 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-xdwxf_a3ebb31f-0f6c-465d-b304-cff9297e8865/registry-server/0.log" Jul 23 12:24:34.455297 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:34.455102 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-ddcdq_4cca4d76-e5f6-466a-ab60-6aae804516c6/manager/0.log" Jul 23 12:24:34.571616 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:34.571583 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-6t9fq_9490542d-4d00-4e2e-83a7-294aba9ab51b/manager/0.log" Jul 23 12:24:36.392705 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.392659 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/alertmanager/0.log" Jul 23 12:24:36.420954 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.420762 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/config-reloader/0.log" Jul 23 12:24:36.449333 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.449296 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/kube-rbac-proxy-web/0.log" Jul 23 12:24:36.474141 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.474090 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/kube-rbac-proxy/0.log" Jul 23 12:24:36.497742 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.497704 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/kube-rbac-proxy-metric/0.log" Jul 23 12:24:36.520912 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.520880 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/prom-label-proxy/0.log" Jul 23 12:24:36.546422 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.546387 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_af2445a8-94ab-4b9e-8a4f-bbba7ebdebb1/init-config-reloader/0.log" Jul 23 12:24:36.674035 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.673954 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vwg66_4474e3ea-0e2d-4f93-97ec-249c45cc9827/kube-state-metrics/0.log" Jul 23 12:24:36.739016 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.738985 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vwg66_4474e3ea-0e2d-4f93-97ec-249c45cc9827/kube-rbac-proxy-main/0.log" Jul 23 12:24:36.801706 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:36.801677 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-vwg66_4474e3ea-0e2d-4f93-97ec-249c45cc9827/kube-rbac-proxy-self/0.log" Jul 23 12:24:37.011576 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.011469 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-npmc7_3b9a134f-b88e-40ca-9749-9d15329fd19e/node-exporter/0.log" Jul 23 12:24:37.034734 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.034703 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-npmc7_3b9a134f-b88e-40ca-9749-9d15329fd19e/kube-rbac-proxy/0.log" Jul 23 12:24:37.063450 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.063420 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-npmc7_3b9a134f-b88e-40ca-9749-9d15329fd19e/init-textfile/0.log" Jul 23 12:24:37.184348 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.184320 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gkzsn_4204af03-fa63-43c4-be7a-e7836115dccd/kube-rbac-proxy-main/0.log" Jul 23 12:24:37.207957 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.207922 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gkzsn_4204af03-fa63-43c4-be7a-e7836115dccd/kube-rbac-proxy-self/0.log" Jul 23 12:24:37.231898 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.231865 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-gkzsn_4204af03-fa63-43c4-be7a-e7836115dccd/openshift-state-metrics/0.log" Jul 23 12:24:37.448501 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.448472 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-ckr8j_c7d4d81b-1b5f-4855-8d2d-c6d88409fd72/prometheus-operator/0.log" Jul 23 12:24:37.469735 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.469703 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-ckr8j_c7d4d81b-1b5f-4855-8d2d-c6d88409fd72/kube-rbac-proxy/0.log" Jul 23 12:24:37.501813 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.501781 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-t5msm_6ecd05d1-08ec-4ac0-b9a8-d50ee7b3ec96/prometheus-operator-admission-webhook/0.log" Jul 23 12:24:37.535600 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.535532 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69b9595b5d-hbd25_0d80a1ac-163a-439b-9d8f-988144eabaf4/telemeter-client/0.log" Jul 23 12:24:37.562605 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.562577 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69b9595b5d-hbd25_0d80a1ac-163a-439b-9d8f-988144eabaf4/reload/0.log" Jul 23 12:24:37.594506 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:37.594466 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69b9595b5d-hbd25_0d80a1ac-163a-439b-9d8f-988144eabaf4/kube-rbac-proxy/0.log" Jul 23 12:24:38.753818 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.753771 2554 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb"] Jul 23 12:24:38.760908 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.760869 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.768065 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.768026 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb"] Jul 23 12:24:38.837575 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.837457 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-lib-modules\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.837575 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.837494 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-proc\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.837869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.837588 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-sys\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.837869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.837691 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9s4t\" (UniqueName: \"kubernetes.io/projected/21332384-57d1-4f31-9d36-268134fdba9a-kube-api-access-d9s4t\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.837869 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.837742 2554 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-podres\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939023 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.938981 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-podres\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939323 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939170 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-podres\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939429 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939403 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-lib-modules\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939581 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939450 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-proc\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939581 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939451 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-lib-modules\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939581 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939519 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-proc\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939581 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939567 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-sys\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939817 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939613 2554 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9s4t\" (UniqueName: \"kubernetes.io/projected/21332384-57d1-4f31-9d36-268134fdba9a-kube-api-access-d9s4t\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.939817 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.939682 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/21332384-57d1-4f31-9d36-268134fdba9a-sys\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:38.948417 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:38.948380 2554 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9s4t\" (UniqueName: \"kubernetes.io/projected/21332384-57d1-4f31-9d36-268134fdba9a-kube-api-access-d9s4t\") pod \"perf-node-gather-daemonset-fnjhb\" (UID: \"21332384-57d1-4f31-9d36-268134fdba9a\") " pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:39.078824 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:39.078772 2554 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:39.292875 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:39.292810 2554 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb"] Jul 23 12:24:39.311541 ip-10-0-141-203 kubenswrapper[2554]: W0723 12:24:39.310073 2554 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod21332384_57d1_4f31_9d36_268134fdba9a.slice/crio-04984e499b6fb528c14c8a272998af13667f5f363749cbc63f21eba6cbdd8ee9 WatchSource:0}: Error finding container 04984e499b6fb528c14c8a272998af13667f5f363749cbc63f21eba6cbdd8ee9: Status 404 returned error can't find the container with id 04984e499b6fb528c14c8a272998af13667f5f363749cbc63f21eba6cbdd8ee9 Jul 23 12:24:39.887263 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:39.887224 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" event={"ID":"21332384-57d1-4f31-9d36-268134fdba9a","Type":"ContainerStarted","Data":"cdc169757d4f0ec1d78d91ba18ac61400d50354266e188293fdf5975bb6f7489"} Jul 23 12:24:39.887690 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:39.887285 2554 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" event={"ID":"21332384-57d1-4f31-9d36-268134fdba9a","Type":"ContainerStarted","Data":"04984e499b6fb528c14c8a272998af13667f5f363749cbc63f21eba6cbdd8ee9"} Jul 23 12:24:39.887690 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:39.887309 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:39.907319 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:39.907256 2554 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" podStartSLOduration=1.907235762 podStartE2EDuration="1.907235762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-23 12:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-23 12:24:39.906703486 +0000 UTC m=+2928.012029917" watchObservedRunningTime="2026-07-23 12:24:39.907235762 +0000 UTC m=+2928.012562174" Jul 23 12:24:40.090915 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:40.090801 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6849845f9-7dhj2_323d76e9-03ed-428b-bffb-09a24eb84243/console/0.log" Jul 23 12:24:41.793221 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:41.793189 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mnkg_0bf927f4-2ca3-486a-8a10-f0609e6a8bcb/dns/0.log" Jul 23 12:24:41.817651 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:41.817600 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7mnkg_0bf927f4-2ca3-486a-8a10-f0609e6a8bcb/kube-rbac-proxy/0.log" Jul 23 12:24:41.960554 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:41.960529 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dx4jr_5cd3fcea-7481-41e8-a0dc-3aad60336adf/dns-node-resolver/0.log" Jul 23 12:24:42.570029 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:42.569999 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-2gqwm_1057adf9-70f7-48f9-a433-c89f628aa741/node-ca/0.log" Jul 23 12:24:43.612379 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:43.612341 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4h2ncb_87931ee1-f888-472f-8be2-98c84ac77bd3/istio-proxy/0.log" Jul 23 12:24:43.898051 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:43.897974 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-zxjwr_3cfd86c0-746c-46e9-a026-487b5d3f81ff/kube-auth-proxy/0.log" Jul 23 12:24:43.930193 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:43.930141 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/0.log" Jul 23 12:24:43.961429 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:43.961388 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/1.log" Jul 23 12:24:44.759876 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:44.759849 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5wqgc_5b1ff801-8f56-491e-a923-93f65cae3c85/serve-healthcheck-canary/0.log" Jul 23 12:24:45.385141 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:45.385109 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b2bcz_9a7ef981-69c7-4308-8032-9f6c186a3159/kube-rbac-proxy/0.log" Jul 23 12:24:45.408588 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:45.408551 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b2bcz_9a7ef981-69c7-4308-8032-9f6c186a3159/exporter/0.log" Jul 23 12:24:45.431572 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:45.431530 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-b2bcz_9a7ef981-69c7-4308-8032-9f6c186a3159/extractor/0.log" Jul 23 12:24:45.906508 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:45.906472 2554 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-5jgf7/perf-node-gather-daemonset-fnjhb" Jul 23 12:24:47.794935 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:47.794908 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-t2k79_778eb9a6-8129-4808-948a-470d451f8304/manager/0.log" Jul 23 12:24:48.072759 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:48.072665 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-wxvg9_b3834800-cac3-4f1d-8b54-670f8f506e64/manager/0.log" Jul 23 12:24:48.209440 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:48.209396 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dlqj6_28b25b2d-cd54-4ac4-8f63-531946cef94e/server/0.log" Jul 23 12:24:48.318084 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:48.318038 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-9cwng_de982565-fe14-4b51-84fa-3ff6144a1617/manager/0.log" Jul 23 12:24:48.344838 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:48.344739 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-n2g77_57c365ab-3761-4a8c-9296-8eda6f086567/manager/0.log" Jul 23 12:24:48.373173 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:48.373143 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c7646c6cc-n2g77_57c365ab-3761-4a8c-9296-8eda6f086567/copy-manifests/0.log" Jul 23 12:24:49.933890 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:49.933857 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-gb65r_06eeb81e-66ac-4662-a5f0-cf9c8fabb332/openshift-lws-operator/0.log" Jul 23 12:24:50.251863 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:50.251772 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4h2ncb_87931ee1-f888-472f-8be2-98c84ac77bd3/istio-proxy/0.log" Jul 23 12:24:50.503322 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:50.503237 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-zxjwr_3cfd86c0-746c-46e9-a026-487b5d3f81ff/kube-auth-proxy/0.log" Jul 23 12:24:50.527967 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:50.527923 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/0.log" Jul 23 12:24:50.561206 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:50.561165 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/1.log" Jul 23 12:24:51.783764 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:51.783733 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4h2ncb_87931ee1-f888-472f-8be2-98c84ac77bd3/istio-proxy/0.log" Jul 23 12:24:52.060377 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:52.060293 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b7f786dc8-zxjwr_3cfd86c0-746c-46e9-a026-487b5d3f81ff/kube-auth-proxy/0.log" Jul 23 12:24:52.089604 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:52.089571 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/0.log" Jul 23 12:24:52.129340 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:52.129304 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-56654dcf7-bxzlp_2d1ea3e1-59ee-4534-8b27-8a054245e1d1/istio-proxy/1.log" Jul 23 12:24:56.529737 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:56.529702 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4rmgk_b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b/kube-multus-additional-cni-plugins/0.log" Jul 23 12:24:56.555349 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:56.555317 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4rmgk_b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b/egress-router-binary-copy/0.log" Jul 23 12:24:56.578211 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:56.578185 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4rmgk_b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b/cni-plugins/0.log" Jul 23 12:24:56.601550 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:56.601520 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4rmgk_b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b/bond-cni-plugin/0.log" Jul 23 12:24:56.630396 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:56.630364 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4rmgk_b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b/routeoverride-cni/0.log" Jul 23 12:24:56.655806 ip-10-0-141-203 kubenswrapper[2554]: I0723 12:24:56.655781 2554 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4rmgk_b2e356d0-8ad6-4d20-8bc6-f99ee8a8555b/whereabouts-cni-bincopy/0.log"