Jul 15 13:04:16.432806 ip-10-0-139-149 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 15 13:04:16.432820 ip-10-0-139-149 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 15 13:04:16.432828 ip-10-0-139-149 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 15 13:04:16.433121 ip-10-0-139-149 systemd[1]: Failed to start Kubernetes Kubelet. Jul 15 13:04:26.682347 ip-10-0-139-149 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 15 13:04:26.682365 ip-10-0-139-149 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot c3b98c2b61cb4b70b99242799218c555 -- Jul 15 13:06:33.925735 ip-10-0-139-149 systemd[1]: Starting Kubernetes Kubelet... Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.345812 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351544 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351555 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351559 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351562 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351565 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351570 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351585 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:34.472625 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351589 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:34.424775 ip-10-0-139-149 systemd[1]: Started Kubernetes Kubelet. Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351592 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351595 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351598 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351601 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351604 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351606 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351609 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351611 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351615 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351617 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351620 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351627 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351630 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351633 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351635 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351638 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351640 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351643 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351645 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:34.473510 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351648 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351651 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351653 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351656 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351659 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351662 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351664 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351667 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351669 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351672 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351674 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351677 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351679 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351682 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351684 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351687 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351691 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351695 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351698 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351701 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:34.474423 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351706 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351709 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351711 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351714 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351717 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351720 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351722 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351725 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351727 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351730 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351733 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351735 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351738 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351741 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351746 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351749 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351752 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351756 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351758 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351761 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:34.475222 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351764 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351766 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351769 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351772 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351774 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351777 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351779 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351782 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351784 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351787 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351790 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351792 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351796 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351798 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351801 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351804 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351806 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351809 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351811 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:34.476028 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351814 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351816 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351820 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351825 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351829 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351834 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351837 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351840 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351842 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351845 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351848 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351851 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.351854 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352581 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352588 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352592 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352595 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352598 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352601 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:34.476690 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352605 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352608 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352613 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352616 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352619 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352621 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352624 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352627 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352630 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352632 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352635 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352638 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352640 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352643 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352645 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352648 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352651 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352653 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352656 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352659 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:34.477327 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352662 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352665 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352667 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352670 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352673 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352675 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352678 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352680 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352683 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352686 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352688 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352691 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352693 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352696 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352700 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352702 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352705 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352707 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352710 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352712 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:34.478035 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352715 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352718 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352720 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352723 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352725 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352728 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352730 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352735 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352738 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352741 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352744 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352748 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352751 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352754 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352756 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352759 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352761 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352764 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352767 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352769 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:34.478712 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352772 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352774 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352778 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352780 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352782 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352785 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352789 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352792 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352794 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352797 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352800 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352802 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352805 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352808 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352811 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352813 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352816 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352818 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352822 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:34.479332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352825 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352828 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352831 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352835 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352838 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352841 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352844 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352846 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352849 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352852 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352854 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352857 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352859 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.352862 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353230 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353234 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353237 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353240 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353243 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353248 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:34.479975 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353250 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353253 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353256 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353258 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353261 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353263 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353266 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353269 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353271 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353274 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353277 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353280 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353282 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353285 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353287 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353290 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353292 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353295 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353297 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:34.480537 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353300 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353303 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353305 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353308 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353311 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353313 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353315 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353318 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353321 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353323 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353326 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353329 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353333 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353335 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353338 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353341 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353343 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353346 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353348 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353351 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:34.481132 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353354 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353357 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353359 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353362 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353365 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353368 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353371 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353373 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353376 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353378 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353381 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353383 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353386 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353388 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353391 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353393 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353396 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353398 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353401 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:34.481783 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353403 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353406 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353408 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353412 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353416 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353420 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353423 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353426 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353429 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353432 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353434 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353437 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353439 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353442 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353445 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353447 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353450 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353452 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353455 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353457 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:34.482374 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353461 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353479 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353483 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353486 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353488 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353491 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353493 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353496 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353498 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353501 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353505 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353508 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353511 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353514 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.353517 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353590 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353597 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353605 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353609 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353613 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 15 13:06:34.482974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353616 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353620 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353624 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353627 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353631 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353634 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353637 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353641 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353644 2564 flags.go:64] FLAG: --cgroup-root="" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353646 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353649 2564 flags.go:64] FLAG: --client-ca-file="" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353652 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353655 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353659 2564 flags.go:64] FLAG: --cluster-domain="" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353662 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353665 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353668 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353671 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353675 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353678 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353681 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353684 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353687 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353690 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 15 13:06:34.483591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353693 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353696 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353699 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353704 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353707 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353711 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353714 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353717 2564 flags.go:64] FLAG: --enable-server="true" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353720 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353724 2564 flags.go:64] FLAG: --event-burst="100" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353728 2564 flags.go:64] FLAG: --event-qps="50" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353731 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353734 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353737 2564 flags.go:64] FLAG: --eviction-hard="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353740 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353743 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353746 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353749 2564 flags.go:64] FLAG: --eviction-soft="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353752 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353755 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353758 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353761 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353764 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353767 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353770 2564 flags.go:64] FLAG: --feature-gates="" Jul 15 13:06:34.484409 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353773 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353776 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353779 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353782 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353785 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353788 2564 flags.go:64] FLAG: --help="false" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353790 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353793 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353796 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353799 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353804 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353807 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353812 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353815 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353818 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353820 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353823 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353826 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353829 2564 flags.go:64] FLAG: --kube-reserved="" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353832 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353835 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353838 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353841 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353844 2564 flags.go:64] FLAG: --lock-file="" Jul 15 13:06:34.485644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353847 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353850 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353853 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353858 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353861 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353864 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353867 2564 flags.go:64] FLAG: --logging-format="text" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353869 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353873 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353875 2564 flags.go:64] FLAG: --manifest-url="" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353878 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353882 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353886 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353889 2564 flags.go:64] FLAG: --max-pods="110" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353892 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353895 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353898 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353901 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353905 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353908 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353912 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353919 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353922 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353925 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 15 13:06:34.489177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353927 2564 flags.go:64] FLAG: --pod-cidr="" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353930 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353936 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353938 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353941 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353945 2564 flags.go:64] FLAG: --port="10250" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353948 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353950 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-03547f0a58000063d" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353954 2564 flags.go:64] FLAG: --qos-reserved="" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353957 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353960 2564 flags.go:64] FLAG: --register-node="true" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353963 2564 flags.go:64] FLAG: --register-with-taints="" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353966 2564 flags.go:64] FLAG: --registry-burst="10" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353969 2564 flags.go:64] FLAG: --registry-qps="5" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353972 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353975 2564 flags.go:64] FLAG: --reserved-memory="" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353978 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353982 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353985 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353987 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353990 2564 flags.go:64] FLAG: --runonce="false" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353993 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353996 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.353999 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354002 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 15 13:06:34.675724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354004 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354009 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354012 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354015 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354021 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354024 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354026 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354029 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354032 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354035 2564 flags.go:64] FLAG: --system-cgroups="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354038 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354043 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354046 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354048 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354052 2564 flags.go:64] FLAG: --tls-min-version="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354055 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354057 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354061 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354064 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354066 2564 flags.go:64] FLAG: --v="2" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354070 2564 flags.go:64] FLAG: --version="false" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354074 2564 flags.go:64] FLAG: --vmodule="" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354077 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.354080 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 15 13:06:34.699832 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354194 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354200 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354204 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354207 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354210 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354213 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354216 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354218 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354221 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354226 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354229 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354231 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354236 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354238 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354241 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354244 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354246 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354249 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354251 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354254 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:34.700583 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354256 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354259 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354261 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354264 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354267 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354269 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354273 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354275 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354278 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354280 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354283 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354285 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354288 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354290 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354293 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354297 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354300 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354304 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354307 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:34.701182 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354309 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354312 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354316 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354319 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354321 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354326 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354328 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354331 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354334 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354337 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354339 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354342 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354344 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354347 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354350 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354352 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354355 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354357 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354360 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:34.701981 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354362 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354365 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354368 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354370 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354373 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354376 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354379 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354381 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354384 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354386 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354389 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354391 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354394 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354396 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354399 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354403 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354405 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354408 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354412 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:34.702877 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354416 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354419 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354422 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354424 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354427 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354430 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354433 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354435 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354438 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354440 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354443 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354445 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354448 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354450 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354453 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354457 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354459 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354462 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354481 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354486 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:34.703572 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354490 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.354517 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.355092 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.362792 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.362806 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.362819 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.362825 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362880 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362885 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362888 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362891 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362894 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362898 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362902 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:34.704231 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362905 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362908 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362911 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362913 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362916 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362919 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362922 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362925 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362927 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362930 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362933 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362936 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362939 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362941 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362945 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362947 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362952 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362955 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362958 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362961 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:34.704886 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362964 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362967 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362970 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362973 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362975 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362978 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362981 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362983 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362986 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362988 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362991 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362994 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362996 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.362999 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363002 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363004 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363006 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363009 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363012 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:34.705561 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363015 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363019 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363022 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363025 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363028 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363031 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363034 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363037 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363040 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363043 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363046 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363049 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363052 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363055 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363057 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363062 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363065 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363067 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363070 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363073 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:34.706175 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363075 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363078 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363080 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363083 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363085 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363088 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363091 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363093 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363096 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363099 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363102 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363104 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363107 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363110 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363112 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363115 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363118 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363121 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363124 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363127 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:34.706779 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363129 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363132 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363135 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363137 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363140 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363142 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363145 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363152 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363154 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363157 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363160 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363162 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363165 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.363170 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:06:34.707389 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363296 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363302 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363306 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363309 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363315 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363317 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363320 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363323 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363326 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363328 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363331 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363334 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363337 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363339 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363342 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363344 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363347 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363351 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363355 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 15 13:06:34.707830 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363359 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363362 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363365 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363368 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363371 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363374 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363378 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363381 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363383 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363386 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363388 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363391 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363393 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363396 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363399 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363401 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363404 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363409 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363412 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363414 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 15 13:06:34.708435 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363417 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363420 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363422 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363425 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363427 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363430 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363433 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363435 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363438 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363441 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363444 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363446 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363449 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363451 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363453 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363456 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363459 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363462 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363478 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363481 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 15 13:06:34.709042 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363492 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363495 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363497 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363500 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363502 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363505 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363508 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363511 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363513 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363517 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363520 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363522 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363525 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363527 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363530 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363532 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363535 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363538 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363541 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363544 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 15 13:06:34.709687 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363547 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363549 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363552 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363555 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363558 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363560 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363563 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363566 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363568 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363571 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363574 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363578 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363580 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363583 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363585 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363588 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363591 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363593 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363596 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 15 13:06:34.710280 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:34.363599 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.363603 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.363701 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.367609 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.369102 2564 server.go:1019] "Starting client certificate rotation" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.369208 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.369246 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.371761 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.375807 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.388830 2564 log.go:25] "Validated CRI v1 runtime API" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.391013 2564 log.go:25] "Validated CRI v1 image API" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.391827 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 15 13:06:34.710921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.395134 2564 fs.go:135] Filesystem UUIDs: map[1bc8ea12-8391-44cd-810d-653c3d646eef:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 83f4f0fd-e250-465c-8462-630f5e3278b9:/dev/nvme0n1p4] Jul 15 13:06:34.711283 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.395148 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.399793 2564 manager.go:216] Machine: {Timestamp:2026-07-15 13:06:34.398641976 +0000 UTC m=+0.354096537 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099846 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2941205b09deb5ffb307f9437ebbb2 SystemUUID:ec294120-5b09-deb5-ffb3-07f9437ebbb2 BootID:c3b98c2b-61cb-4b70-b992-42799218c555 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:1f:31:18:40:d5 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:1f:31:18:40:d5 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:36:a4:fe:59:67:b6 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.399885 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.399983 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.400996 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.401018 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-149.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.401709 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.401719 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 15 13:06:34.885196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.401726 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.402440 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.403231 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.404333 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.405981 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406090 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406123 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406134 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406145 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406153 2564 kubelet.go:398] "Adding apiserver pod source" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406161 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406420 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.406434 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.414781 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.416283 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.416310 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417510 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417524 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417530 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417536 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417543 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417549 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417556 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417562 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417568 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417574 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417582 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.417587 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.419064 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.419072 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.420491 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.420493 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.422072 2564 server.go:1268] "Started kubelet" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.422086 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-149.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.422140 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.422144 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.427094 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.427926 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.428159 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 15 13:06:34.887974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.431895 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cw9fj" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.431369 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-149.ec2.internal.18c27868ebea075b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-149.ec2.internal,UID:ip-10-0-139-149.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-149.ec2.internal,},FirstTimestamp:2026-07-15 13:06:34.422052699 +0000 UTC m=+0.377507253,LastTimestamp:2026-07-15 13:06:34.422052699 +0000 UTC m=+0.377507253,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-149.ec2.internal,}" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.432819 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.433283 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.433282 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.433372 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.433910 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.433909 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.433931 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434068 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434077 2564 factory.go:55] Registering systemd factory Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434080 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434092 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434083 2564 factory.go:223] Registration of the systemd container factory successfully Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434280 2564 factory.go:153] Registering CRI-O factory Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434294 2564 factory.go:223] Registration of the crio container factory successfully Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434312 2564 factory.go:103] Registering Raw factory Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.434321 2564 manager.go:1196] Started watching for new ooms in manager Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.435167 2564 manager.go:349] Starting recovery of all containers Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.443648 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cw9fj" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.447271 2564 manager.go:354] Recovery completed Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.448084 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.448128 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-149.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.452463 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.454183 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.454203 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.454213 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.454788 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.454796 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.454810 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.456087 2564 policy_none.go:49] "None policy: Start" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.456100 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.456109 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 15 13:06:34.889228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.456342 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.457181 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.457209 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.457220 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.457424 2564 policy_none.go:47] "Start" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498363 2564 manager.go:337] "Starting Device Plugin manager" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.498390 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498401 2564 server.go:87] "Starting device plugin registration server" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498647 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498662 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498739 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498829 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.498839 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.499622 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.499675 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.601646 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.603109 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.603141 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.603152 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.603177 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.615866 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.615886 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-149.ec2.internal\": node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.640911 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.649421 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.650625 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.650659 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.650679 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.650708 2564 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.653111 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.741172 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.751005 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal"] Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.751092 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.752021 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.752051 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.752061 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.753544 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.753672 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.890306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.753697 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.754546 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.754564 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.754574 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.754776 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.754787 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.754796 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.756423 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.756443 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.757156 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientMemory" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.757173 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasNoDiskPressure" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.757182 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeHasSufficientPID" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.793308 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-149.ec2.internal\" not found" node="ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.796847 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-149.ec2.internal\" not found" node="ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.837928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e57446bdf2328c4489e6db9a2e20979c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal\" (UID: \"e57446bdf2328c4489e6db9a2e20979c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.837986 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e57446bdf2328c4489e6db9a2e20979c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal\" (UID: \"e57446bdf2328c4489e6db9a2e20979c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.838004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dd7a4f18fc1aca27c6a9caeb4dafb15-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.ec2.internal\" (UID: \"0dd7a4f18fc1aca27c6a9caeb4dafb15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" Jul 15 13:06:34.891362 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.841406 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.939140 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e57446bdf2328c4489e6db9a2e20979c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal\" (UID: \"e57446bdf2328c4489e6db9a2e20979c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.939169 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e57446bdf2328c4489e6db9a2e20979c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal\" (UID: \"e57446bdf2328c4489e6db9a2e20979c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.939195 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dd7a4f18fc1aca27c6a9caeb4dafb15-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.ec2.internal\" (UID: \"0dd7a4f18fc1aca27c6a9caeb4dafb15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.939221 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0dd7a4f18fc1aca27c6a9caeb4dafb15-config\") pod \"kube-apiserver-proxy-ip-10-0-139-149.ec2.internal\" (UID: \"0dd7a4f18fc1aca27c6a9caeb4dafb15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.939239 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e57446bdf2328c4489e6db9a2e20979c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal\" (UID: \"e57446bdf2328c4489e6db9a2e20979c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:34.939247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e57446bdf2328c4489e6db9a2e20979c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal\" (UID: \"e57446bdf2328c4489e6db9a2e20979c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:34.942491 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.085122 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.043211 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.268075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.096584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.268075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.099357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.268075 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.143916 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.268075 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.244627 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.450945 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.345649 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.450945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.369563 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 15 13:06:35.450945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.369691 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 15 13:06:35.450945 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.445722 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.450945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.447965 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-14 13:01:35 +0000 UTC" deadline="2028-01-30 00:11:21.656684815 +0000 UTC" Jul 15 13:06:35.450945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.447990 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13523h4m46.208698093s" Jul 15 13:06:35.630262 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.492228 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:35.630262 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.546189 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.806071 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.646647 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.806071 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:35.747334 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-149.ec2.internal\" not found" Jul 15 13:06:35.806071 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:35.773863 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57446bdf2328c4489e6db9a2e20979c.slice/crio-e942e8702bf6063d6d4c1698d81ce64d1b6f69d2a18f590bd0464d7fee71c069 WatchSource:0}: Error finding container e942e8702bf6063d6d4c1698d81ce64d1b6f69d2a18f590bd0464d7fee71c069: Status 404 returned error can't find the container with id e942e8702bf6063d6d4c1698d81ce64d1b6f69d2a18f590bd0464d7fee71c069 Jul 15 13:06:35.806071 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:35.774155 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd7a4f18fc1aca27c6a9caeb4dafb15.slice/crio-3ac9d8ed035da4f1f48d99732d25915f90074e7ca2bee4723a83aa908d4f19a9 WatchSource:0}: Error finding container 3ac9d8ed035da4f1f48d99732d25915f90074e7ca2bee4723a83aa908d4f19a9: Status 404 returned error can't find the container with id 3ac9d8ed035da4f1f48d99732d25915f90074e7ca2bee4723a83aa908d4f19a9 Jul 15 13:06:35.806071 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.774944 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:35.806071 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.778866 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:06:35.971768 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.833695 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.971768 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.847973 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 13:06:35.971768 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.850038 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" Jul 15 13:06:35.971768 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.859109 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 15 13:06:35.971768 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:35.870199 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:36.483329 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.483302 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 15 13:06:36.483329 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.483318 2564 apiserver.go:52] "Watching apiserver" Jul 15 13:06:36.495814 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.495787 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 15 13:06:36.496275 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.496249 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-spqmk","openshift-multus/multus-nwkk4","openshift-network-diagnostics/network-check-target-vtkg6","openshift-network-operator/iptables-alerter-b548v","openshift-ovn-kubernetes/ovnkube-node-ghhfs","kube-system/global-pull-secret-syncer-lgnvn","kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal","openshift-cluster-node-tuning-operator/tuned-grrjx","openshift-dns/node-resolver-c4wgv","openshift-image-registry/node-ca-x7jkc","openshift-multus/network-metrics-daemon-5vjbj","kube-system/konnectivity-agent-jgwwr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal"] Jul 15 13:06:36.497907 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.497885 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.500035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.500007 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.501314 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.501295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:36.501422 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.501401 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:36.501759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.501740 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qst2t\"" Jul 15 13:06:36.502033 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.502014 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.502267 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.502250 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.502846 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.502612 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 15 13:06:36.502846 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.502643 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.502846 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.502710 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 15 13:06:36.503061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.503042 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-s8bqx\"" Jul 15 13:06:36.503302 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.503283 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.503500 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.503284 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.504271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.504247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.505093 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.505017 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 15 13:06:36.505764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.505745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.505860 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.505814 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:36.507641 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.507621 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.509011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.508982 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.509367 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.509340 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 15 13:06:36.509571 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.509549 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.509659 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.509601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 15 13:06:36.509941 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.509908 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.510755 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.510721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f2z2q\"" Jul 15 13:06:36.510911 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.510891 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.511193 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.510734 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 15 13:06:36.511695 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.511673 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bdv4c\"" Jul 15 13:06:36.511806 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.511679 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.511806 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.511752 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-nm2tj\"" Jul 15 13:06:36.511806 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.511804 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 15 13:06:36.512008 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.511894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.512008 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.511942 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.512119 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.512010 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 15 13:06:36.512395 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.512370 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.512503 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.512436 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.512573 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.512511 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.512736 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.512716 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.513110 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.513091 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 15 13:06:36.513426 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.513253 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8j92l\"" Jul 15 13:06:36.513426 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.513372 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k77dx\"" Jul 15 13:06:36.514533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.514471 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 15 13:06:36.514627 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.514606 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 15 13:06:36.514730 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.514714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:36.514810 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.514783 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:36.514887 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.514871 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 15 13:06:36.516282 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.516262 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.516379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.516322 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 15 13:06:36.517734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.517716 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 15 13:06:36.517833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.517720 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-8dvgg\"" Jul 15 13:06:36.518992 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.518972 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 15 13:06:36.519139 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.519122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-45nvz\"" Jul 15 13:06:36.519196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.518977 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 15 13:06:36.535107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.535083 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kc2gs" Jul 15 13:06:36.535604 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.535587 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 15 13:06:36.543003 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.542983 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kc2gs" Jul 15 13:06:36.549089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.549168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysctl-d\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.549168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-cnibin\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.549168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-slash\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549171 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysctl-conf\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549190 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-tuned\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549206 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-kubelet\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-daemon-config\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549261 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-cni-binary-copy\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl98z\" (UniqueName: \"kubernetes.io/projected/03957cc5-0ec0-48c6-a682-ab1a0d26a273-kube-api-access-sl98z\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.549331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549319 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-registration-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-system-cni-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549367 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-system-cni-dir\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a216b80-147e-4f1e-b0cc-a001496017c0-host-slash\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-etc-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549513 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-run-ovn-kubernetes\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-cni-bin\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-modprobe-d\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-kubernetes\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549650 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-kubelet\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549675 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28d22\" (UniqueName: \"kubernetes.io/projected/6eead1c0-e669-4f42-be4a-ad182648b50c-kube-api-access-28d22\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.549701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-cnibin\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-k8s-cni-cncf-io\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-hostroot\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03957cc5-0ec0-48c6-a682-ab1a0d26a273-serviceca\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-systemd-units\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549868 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-node-log\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549907 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdtt8\" (UniqueName: \"kubernetes.io/projected/c5b39e97-5dd3-4472-b87f-436050a3656e-kube-api-access-jdtt8\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.549979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-netns\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550020 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-sys-fs\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.550594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550072 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysconfig\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.550594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550568 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-lib-modules\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.550695 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c5b39e97-5dd3-4472-b87f-436050a3656e-hosts-file\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.550695 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-ovnkube-script-lib\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.550695 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dln7j\" (UniqueName: \"kubernetes.io/projected/2c444ed9-604e-4fae-8437-7b0548983837-kube-api-access-dln7j\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.550827 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550692 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-run\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.550827 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-os-release\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.550827 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c444ed9-604e-4fae-8437-7b0548983837-ovn-node-metrics-cert\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.550827 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e85af3-fa0a-40e5-9350-2133b1797824-kubelet-config\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.551011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550800 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-sys\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.551011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-cni-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.551011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-multus-certs\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.551011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:36.551011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550949 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-ovnkube-config\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.551011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.550976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-etc-kubernetes\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551023 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgssj\" (UniqueName: \"kubernetes.io/projected/8a216b80-147e-4f1e-b0cc-a001496017c0-kube-api-access-fgssj\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551099 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-ovn\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551133 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7ctz\" (UniqueName: \"kubernetes.io/projected/0d4460e2-5dfa-42ab-9921-6a617c2d207d-kube-api-access-f7ctz\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551158 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03957cc5-0ec0-48c6-a682-ab1a0d26a273-host\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551202 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-var-lib-kubelet\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.551243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-host\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eead1c0-e669-4f42-be4a-ad182648b50c-tmp\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551343 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-os-release\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551394 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-systemd\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551420 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-log-socket\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551442 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-env-overrides\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33145dcd-966e-41d4-b9e7-b0c1dfe16528-agent-certs\") pod \"konnectivity-agent-jgwwr\" (UID: \"33145dcd-966e-41d4-b9e7-b0c1dfe16528\") " pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33145dcd-966e-41d4-b9e7-b0c1dfe16528-konnectivity-ca\") pod \"konnectivity-agent-jgwwr\" (UID: \"33145dcd-966e-41d4-b9e7-b0c1dfe16528\") " pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-tuning-conf-dir\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551566 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-socket-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:36.551616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551614 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551639 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-socket-dir-parent\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-cni-bin\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551684 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-conf-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551709 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zk2s\" (UniqueName: \"kubernetes.io/projected/6349c6b9-5951-4fa4-97b5-6d0497b68807-kube-api-access-5zk2s\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551734 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-run-netns\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551771 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b39e97-5dd3-4472-b87f-436050a3656e-tmp-dir\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551834 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-cni-multus\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551874 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljzsl\" (UniqueName: \"kubernetes.io/projected/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-kube-api-access-ljzsl\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551956 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28rfm\" (UniqueName: \"kubernetes.io/projected/e70c5403-e894-47ff-9b9f-c32be099611a-kube-api-access-28rfm\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.551983 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-cni-netd\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552009 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.552264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552032 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-systemd\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.552948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d4460e2-5dfa-42ab-9921-6a617c2d207d-cni-binary-copy\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.552948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-device-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.552948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a216b80-147e-4f1e-b0cc-a001496017c0-iptables-alerter-script\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.552948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552121 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-var-lib-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.552948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.552141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e85af3-fa0a-40e5-9350-2133b1797824-dbus\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.639643 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.639608 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 15 13:06:36.652694 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-os-release\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.652694 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c444ed9-604e-4fae-8437-7b0548983837-ovn-node-metrics-cert\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.652876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652715 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e85af3-fa0a-40e5-9350-2133b1797824-kubelet-config\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.652876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-sys\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.652876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652762 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-cni-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.652876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652807 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-multus-certs\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.652876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652858 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-ovnkube-config\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/18e85af3-fa0a-40e5-9350-2133b1797824-kubelet-config\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.652930 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-sys\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653003 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-etc-kubernetes\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-multus-certs\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.653061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgssj\" (UniqueName: \"kubernetes.io/projected/8a216b80-147e-4f1e-b0cc-a001496017c0-kube-api-access-fgssj\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653090 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-ovn\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7ctz\" (UniqueName: \"kubernetes.io/projected/0d4460e2-5dfa-42ab-9921-6a617c2d207d-kube-api-access-f7ctz\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653165 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.653183 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:06:37.153127027 +0000 UTC m=+3.108581580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653221 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03957cc5-0ec0-48c6-a682-ab1a0d26a273-host\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-var-lib-kubelet\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-host\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653310 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eead1c0-e669-4f42-be4a-ad182648b50c-tmp\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-os-release\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-systemd\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.653410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653390 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-log-socket\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653420 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-env-overrides\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653420 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-cni-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.652828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-os-release\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653464 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-ovn\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653500 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33145dcd-966e-41d4-b9e7-b0c1dfe16528-agent-certs\") pod \"konnectivity-agent-jgwwr\" (UID: \"33145dcd-966e-41d4-b9e7-b0c1dfe16528\") " pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653520 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-etc-kubernetes\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33145dcd-966e-41d4-b9e7-b0c1dfe16528-konnectivity-ca\") pod \"konnectivity-agent-jgwwr\" (UID: \"33145dcd-966e-41d4-b9e7-b0c1dfe16528\") " pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/03957cc5-0ec0-48c6-a682-ab1a0d26a273-host\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653577 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-tuning-conf-dir\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653613 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-socket-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-var-lib-kubelet\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-systemd\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653644 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-log-socket\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-run-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653783 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-socket-dir-parent\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-host\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-tuning-conf-dir\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653633 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-ovnkube-config\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.653721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-socket-dir-parent\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-cni-bin\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-socket-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654100 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-conf-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654134 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zk2s\" (UniqueName: \"kubernetes.io/projected/6349c6b9-5951-4fa4-97b5-6d0497b68807-kube-api-access-5zk2s\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654139 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-conf-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-run-netns\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-env-overrides\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.654803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b39e97-5dd3-4472-b87f-436050a3656e-tmp-dir\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654175 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-cni-bin\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654225 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-cni-multus\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-os-release\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654258 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljzsl\" (UniqueName: \"kubernetes.io/projected/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-kube-api-access-ljzsl\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654286 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28rfm\" (UniqueName: \"kubernetes.io/projected/e70c5403-e894-47ff-9b9f-c32be099611a-kube-api-access-28rfm\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-cni-netd\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-systemd\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654426 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5b39e97-5dd3-4472-b87f-436050a3656e-tmp-dir\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654441 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d4460e2-5dfa-42ab-9921-6a617c2d207d-cni-binary-copy\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654469 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-device-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654489 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-cni-multus\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.655589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a216b80-147e-4f1e-b0cc-a001496017c0-iptables-alerter-script\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654523 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-var-lib-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654570 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e85af3-fa0a-40e5-9350-2133b1797824-dbus\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654600 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-systemd\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654636 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysctl-d\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654684 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-cnibin\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysctl-d\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-slash\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysctl-conf\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654822 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-device-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-tuned\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654860 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-cni-netd\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654902 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-var-lib-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656285 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.655024 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.655091 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:37.15507269 +0000 UTC m=+3.110527466 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-run-netns\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-etc-selinux\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655164 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-slash\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655206 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-cnibin\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.654770 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655273 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/18e85af3-fa0a-40e5-9350-2133b1797824-dbus\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655326 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0d4460e2-5dfa-42ab-9921-6a617c2d207d-cni-binary-copy\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655359 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysctl-conf\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-kubelet\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-daemon-config\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-cni-binary-copy\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.656774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655828 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl98z\" (UniqueName: \"kubernetes.io/projected/03957cc5-0ec0-48c6-a682-ab1a0d26a273-kube-api-access-sl98z\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-registration-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655882 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-system-cni-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-system-cni-dir\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a216b80-147e-4f1e-b0cc-a001496017c0-host-slash\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655958 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-etc-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655957 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/33145dcd-966e-41d4-b9e7-b0c1dfe16528-konnectivity-ca\") pod \"konnectivity-agent-jgwwr\" (UID: \"33145dcd-966e-41d4-b9e7-b0c1dfe16528\") " pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.655991 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-run-ovn-kubernetes\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-cni-bin\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656086 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-modprobe-d\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-kubernetes\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656117 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-registration-dir\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656139 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-var-lib-kubelet\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.657425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-kubelet\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28d22\" (UniqueName: \"kubernetes.io/projected/6eead1c0-e669-4f42-be4a-ad182648b50c-kube-api-access-28d22\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656281 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-cnibin\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-k8s-cni-cncf-io\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656345 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-hostroot\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-cni-bin\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656407 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03957cc5-0ec0-48c6-a682-ab1a0d26a273-serviceca\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-systemd-units\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-system-cni-dir\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656570 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6349c6b9-5951-4fa4-97b5-6d0497b68807-system-cni-dir\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8a216b80-147e-4f1e-b0cc-a001496017c0-host-slash\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656628 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-node-log\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdtt8\" (UniqueName: \"kubernetes.io/projected/c5b39e97-5dd3-4472-b87f-436050a3656e-kube-api-access-jdtt8\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-netns\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656714 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-kubernetes\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.658130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656809 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/0d4460e2-5dfa-42ab-9921-6a617c2d207d-multus-daemon-config\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-kubelet\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-cnibin\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.656992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-k8s-cni-cncf-io\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-hostroot\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657235 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-sys-fs\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657259 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysconfig\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657282 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-lib-modules\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657306 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c5b39e97-5dd3-4472-b87f-436050a3656e-hosts-file\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-ovnkube-script-lib\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657346 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-cni-binary-copy\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657360 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dln7j\" (UniqueName: \"kubernetes.io/projected/2c444ed9-604e-4fae-8437-7b0548983837-kube-api-access-dln7j\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-run\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657458 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6eead1c0-e669-4f42-be4a-ad182648b50c-tmp\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.658805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-modprobe-d\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657603 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/03957cc5-0ec0-48c6-a682-ab1a0d26a273-serviceca\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-run\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657685 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-etc-openvswitch\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e70c5403-e894-47ff-9b9f-c32be099611a-sys-fs\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657720 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-host-run-ovn-kubernetes\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657726 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/2c444ed9-604e-4fae-8437-7b0548983837-ovn-node-metrics-cert\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c5b39e97-5dd3-4472-b87f-436050a3656e-hosts-file\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657972 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-systemd-units\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.657978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0d4460e2-5dfa-42ab-9921-6a617c2d207d-host-run-netns\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.658032 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-lib-modules\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.658037 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-sysconfig\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.658084 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/2c444ed9-604e-4fae-8437-7b0548983837-node-log\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.658218 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6349c6b9-5951-4fa4-97b5-6d0497b68807-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.659356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.658360 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/2c444ed9-604e-4fae-8437-7b0548983837-ovnkube-script-lib\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.660034 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.658880 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/6eead1c0-e669-4f42-be4a-ad182648b50c-etc-tuned\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.660034 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.659887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/33145dcd-966e-41d4-b9e7-b0c1dfe16528-agent-certs\") pod \"konnectivity-agent-jgwwr\" (UID: \"33145dcd-966e-41d4-b9e7-b0c1dfe16528\") " pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:36.660890 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.660842 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" event={"ID":"0dd7a4f18fc1aca27c6a9caeb4dafb15","Type":"ContainerStarted","Data":"3ac9d8ed035da4f1f48d99732d25915f90074e7ca2bee4723a83aa908d4f19a9"} Jul 15 13:06:36.662513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.662436 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" event={"ID":"e57446bdf2328c4489e6db9a2e20979c","Type":"ContainerStarted","Data":"e942e8702bf6063d6d4c1698d81ce64d1b6f69d2a18f590bd0464d7fee71c069"} Jul 15 13:06:36.663724 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.663693 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:36.663724 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.663724 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:36.663888 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.663738 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:36.663888 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:36.663813 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:37.16379891 +0000 UTC m=+3.119253475 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:36.666244 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.666063 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7ctz\" (UniqueName: \"kubernetes.io/projected/0d4460e2-5dfa-42ab-9921-6a617c2d207d-kube-api-access-f7ctz\") pod \"multus-nwkk4\" (UID: \"0d4460e2-5dfa-42ab-9921-6a617c2d207d\") " pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.666986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.666933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28rfm\" (UniqueName: \"kubernetes.io/projected/e70c5403-e894-47ff-9b9f-c32be099611a-kube-api-access-28rfm\") pod \"aws-ebs-csi-driver-node-8b4lv\" (UID: \"e70c5403-e894-47ff-9b9f-c32be099611a\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.666986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.666950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljzsl\" (UniqueName: \"kubernetes.io/projected/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-kube-api-access-ljzsl\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:36.667138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.667087 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8a216b80-147e-4f1e-b0cc-a001496017c0-iptables-alerter-script\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.668023 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.667994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zk2s\" (UniqueName: \"kubernetes.io/projected/6349c6b9-5951-4fa4-97b5-6d0497b68807-kube-api-access-5zk2s\") pod \"multus-additional-cni-plugins-spqmk\" (UID: \"6349c6b9-5951-4fa4-97b5-6d0497b68807\") " pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.668430 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.668410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgssj\" (UniqueName: \"kubernetes.io/projected/8a216b80-147e-4f1e-b0cc-a001496017c0-kube-api-access-fgssj\") pod \"iptables-alerter-b548v\" (UID: \"8a216b80-147e-4f1e-b0cc-a001496017c0\") " pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.669650 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.669631 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdtt8\" (UniqueName: \"kubernetes.io/projected/c5b39e97-5dd3-4472-b87f-436050a3656e-kube-api-access-jdtt8\") pod \"node-resolver-c4wgv\" (UID: \"c5b39e97-5dd3-4472-b87f-436050a3656e\") " pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.670158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.670135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28d22\" (UniqueName: \"kubernetes.io/projected/6eead1c0-e669-4f42-be4a-ad182648b50c-kube-api-access-28d22\") pod \"tuned-grrjx\" (UID: \"6eead1c0-e669-4f42-be4a-ad182648b50c\") " pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.671108 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.671071 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl98z\" (UniqueName: \"kubernetes.io/projected/03957cc5-0ec0-48c6-a682-ab1a0d26a273-kube-api-access-sl98z\") pod \"node-ca-x7jkc\" (UID: \"03957cc5-0ec0-48c6-a682-ab1a0d26a273\") " pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.671522 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.671502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dln7j\" (UniqueName: \"kubernetes.io/projected/2c444ed9-604e-4fae-8437-7b0548983837-kube-api-access-dln7j\") pod \"ovnkube-node-ghhfs\" (UID: \"2c444ed9-604e-4fae-8437-7b0548983837\") " pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.810463 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.810352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-x7jkc" Jul 15 13:06:36.819857 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.819829 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" Jul 15 13:06:36.829784 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.829756 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-b548v" Jul 15 13:06:36.835655 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.835624 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:06:36.842407 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.842391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-c4wgv" Jul 15 13:06:36.850347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.850328 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-grrjx" Jul 15 13:06:36.858151 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.858129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nwkk4" Jul 15 13:06:36.866284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.866265 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-spqmk" Jul 15 13:06:36.871997 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:36.871975 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:06:37.085108 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:37.085078 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03957cc5_0ec0_48c6_a682_ab1a0d26a273.slice/crio-556e8c57dac5c73060fe83187fa8565562339562e709e9fcede3ab5e8bfc21eb WatchSource:0}: Error finding container 556e8c57dac5c73060fe83187fa8565562339562e709e9fcede3ab5e8bfc21eb: Status 404 returned error can't find the container with id 556e8c57dac5c73060fe83187fa8565562339562e709e9fcede3ab5e8bfc21eb Jul 15 13:06:37.086617 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:37.086589 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a216b80_147e_4f1e_b0cc_a001496017c0.slice/crio-6c6c5a49ef57a722c517857a9b7d132722fe52ebdf5e8dbd05d2e25ea5f8287c WatchSource:0}: Error finding container 6c6c5a49ef57a722c517857a9b7d132722fe52ebdf5e8dbd05d2e25ea5f8287c: Status 404 returned error can't find the container with id 6c6c5a49ef57a722c517857a9b7d132722fe52ebdf5e8dbd05d2e25ea5f8287c Jul 15 13:06:37.087467 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:37.087435 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33145dcd_966e_41d4_b9e7_b0c1dfe16528.slice/crio-9d2f3d5ce00b92c6da5f38e5078af227d1ea3196ac022626ab5d255c6921b993 WatchSource:0}: Error finding container 9d2f3d5ce00b92c6da5f38e5078af227d1ea3196ac022626ab5d255c6921b993: Status 404 returned error can't find the container with id 9d2f3d5ce00b92c6da5f38e5078af227d1ea3196ac022626ab5d255c6921b993 Jul 15 13:06:37.088310 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:37.088191 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6349c6b9_5951_4fa4_97b5_6d0497b68807.slice/crio-bc96845c3e50bbb538d3f1dfcc6244b14657ba2b9f6600bc264d45d57aa3137f WatchSource:0}: Error finding container bc96845c3e50bbb538d3f1dfcc6244b14657ba2b9f6600bc264d45d57aa3137f: Status 404 returned error can't find the container with id bc96845c3e50bbb538d3f1dfcc6244b14657ba2b9f6600bc264d45d57aa3137f Jul 15 13:06:37.089030 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:37.089004 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c444ed9_604e_4fae_8437_7b0548983837.slice/crio-397f46b4743aa966019c40ae69f948fc78108a1af95381c7a5aa7617bb69142a WatchSource:0}: Error finding container 397f46b4743aa966019c40ae69f948fc78108a1af95381c7a5aa7617bb69142a: Status 404 returned error can't find the container with id 397f46b4743aa966019c40ae69f948fc78108a1af95381c7a5aa7617bb69142a Jul 15 13:06:37.092064 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:06:37.092044 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d4460e2_5dfa_42ab_9921_6a617c2d207d.slice/crio-112c201e96bbe8ace9cdaf82a8d5d8d00ba39038368be25a638946fe5ca4e8ba WatchSource:0}: Error finding container 112c201e96bbe8ace9cdaf82a8d5d8d00ba39038368be25a638946fe5ca4e8ba: Status 404 returned error can't find the container with id 112c201e96bbe8ace9cdaf82a8d5d8d00ba39038368be25a638946fe5ca4e8ba Jul 15 13:06:37.162050 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.162020 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:37.162212 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.162098 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:37.162212 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.162181 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:37.162212 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.162191 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:37.162371 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.162244 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:38.162229534 +0000 UTC m=+4.117684077 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:37.162371 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.162259 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:06:38.16225162 +0000 UTC m=+4.117706161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:37.263098 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.263063 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:37.263274 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.263256 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:37.263323 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.263281 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:37.263323 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.263292 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:37.263389 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.263347 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:38.263332899 +0000 UTC m=+4.218787442 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:37.545163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.545116 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 13:01:37 +0000 UTC" deadline="2028-04-21 01:42:11.21010535 +0000 UTC" Jul 15 13:06:37.545163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.545160 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15492h35m33.664950794s" Jul 15 13:06:37.651640 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.651563 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:37.652493 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:37.651706 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:37.686639 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.686476 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-grrjx" event={"ID":"6eead1c0-e669-4f42-be4a-ad182648b50c","Type":"ContainerStarted","Data":"a4127ba6af7dff3029a8a0e4f9074fe2c044e3e533a6da29de12e2c4e4b42d6e"} Jul 15 13:06:37.694725 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.694691 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x7jkc" event={"ID":"03957cc5-0ec0-48c6-a682-ab1a0d26a273","Type":"ContainerStarted","Data":"556e8c57dac5c73060fe83187fa8565562339562e709e9fcede3ab5e8bfc21eb"} Jul 15 13:06:37.714547 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.714441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c4wgv" event={"ID":"c5b39e97-5dd3-4472-b87f-436050a3656e","Type":"ContainerStarted","Data":"e807b26520535883b35398a08e56067f6c164e549e689330ca14473ad88291e7"} Jul 15 13:06:37.726387 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.726273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" event={"ID":"e70c5403-e894-47ff-9b9f-c32be099611a","Type":"ContainerStarted","Data":"2b36ebb3eefe1ae8c845cd04dd2c837872f70f8b2bed72986b11e91f801a81a3"} Jul 15 13:06:37.742295 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.742225 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwkk4" event={"ID":"0d4460e2-5dfa-42ab-9921-6a617c2d207d","Type":"ContainerStarted","Data":"112c201e96bbe8ace9cdaf82a8d5d8d00ba39038368be25a638946fe5ca4e8ba"} Jul 15 13:06:37.745387 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.745356 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"397f46b4743aa966019c40ae69f948fc78108a1af95381c7a5aa7617bb69142a"} Jul 15 13:06:37.748823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.748789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerStarted","Data":"bc96845c3e50bbb538d3f1dfcc6244b14657ba2b9f6600bc264d45d57aa3137f"} Jul 15 13:06:37.753512 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.753441 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jgwwr" event={"ID":"33145dcd-966e-41d4-b9e7-b0c1dfe16528","Type":"ContainerStarted","Data":"9d2f3d5ce00b92c6da5f38e5078af227d1ea3196ac022626ab5d255c6921b993"} Jul 15 13:06:37.757871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.757798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b548v" event={"ID":"8a216b80-147e-4f1e-b0cc-a001496017c0","Type":"ContainerStarted","Data":"6c6c5a49ef57a722c517857a9b7d132722fe52ebdf5e8dbd05d2e25ea5f8287c"} Jul 15 13:06:37.769510 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.769426 2564 generic.go:356] "Generic (PLEG): container finished" podID="e57446bdf2328c4489e6db9a2e20979c" containerID="3a6a8a3a7a39f3a636da9674887ff6a01d068031e7490c8daf48bf9eb34098b2" exitCode=0 Jul 15 13:06:37.769510 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:37.769462 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" event={"ID":"e57446bdf2328c4489e6db9a2e20979c","Type":"ContainerDied","Data":"3a6a8a3a7a39f3a636da9674887ff6a01d068031e7490c8daf48bf9eb34098b2"} Jul 15 13:06:38.173237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.172606 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:38.173237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.172910 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:38.173237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.172951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:38.173237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.172993 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:40.17296859 +0000 UTC m=+6.128423155 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:38.173237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.173082 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:38.173237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.173140 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:06:40.173121763 +0000 UTC m=+6.128576330 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:38.273481 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.273364 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:38.273663 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.273576 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:38.273663 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.273608 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:38.273663 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.273623 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:38.273849 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.273693 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:40.273670282 +0000 UTC m=+6.229124855 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:38.546135 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.546069 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-14 13:01:37 +0000 UTC" deadline="2028-01-21 18:44:27.016727751 +0000 UTC" Jul 15 13:06:38.546248 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.546136 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13325h37m48.470597577s" Jul 15 13:06:38.652012 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.651454 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:38.652012 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.651624 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:38.652012 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.651635 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:38.652012 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:38.651770 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:38.781825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.781789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" event={"ID":"e57446bdf2328c4489e6db9a2e20979c","Type":"ContainerStarted","Data":"99bafdfa7227f0ae3e94470602ae3ede312321c21d3510b805c8de0b06659e03"} Jul 15 13:06:38.798110 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:38.798013 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-149.ec2.internal" podStartSLOduration=2.797969658 podStartE2EDuration="2.797969658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:38.797270677 +0000 UTC m=+4.752725244" watchObservedRunningTime="2026-07-15 13:06:38.797969658 +0000 UTC m=+4.753424227" Jul 15 13:06:39.651854 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:39.651818 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:39.652029 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:39.651960 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:40.192009 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:40.191964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:40.192205 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:40.192059 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:40.192205 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.192137 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:40.192205 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.192179 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:40.192376 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.192219 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:06:44.19219591 +0000 UTC m=+10.147650454 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:40.192376 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.192239 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:44.192229542 +0000 UTC m=+10.147684087 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:40.293210 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:40.293071 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:40.293390 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.293259 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:40.293390 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.293286 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:40.293390 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.293300 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:40.293390 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.293364 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:44.293343438 +0000 UTC m=+10.248797983 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:40.656163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:40.655432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:40.656163 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.655578 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:40.656163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:40.655951 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:40.656163 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:40.656053 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:41.652084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:41.651561 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:41.652084 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:41.651719 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:42.655803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:42.655718 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:42.656269 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:42.655848 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:42.656269 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:42.655943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:42.656269 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:42.656042 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:43.650976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:43.650925 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:43.651177 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:43.651076 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:44.230760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:44.230720 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:44.231172 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:44.230798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:44.231172 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.230955 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:44.231172 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.231019 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:06:52.230999041 +0000 UTC m=+18.186453592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:44.231444 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.231424 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:44.231567 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.231481 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:52.231464846 +0000 UTC m=+18.186919393 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:44.331882 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:44.331841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:44.332077 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.332059 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:44.332170 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.332081 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:44.332170 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.332096 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:44.332170 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.332162 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:06:52.332139209 +0000 UTC m=+18.287593754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:44.652778 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:44.652192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:44.652778 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.652307 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:44.653002 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:44.652828 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:44.653002 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:44.652947 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:45.651236 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:45.651203 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:45.651701 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:45.651334 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:46.654875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:46.654193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:46.654875 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:46.654316 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:46.654875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:46.654727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:46.654875 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:46.654823 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:47.651293 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:47.651275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:47.651385 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:47.651367 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:48.651741 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:48.651689 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:48.652195 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:48.651821 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:48.652195 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:48.651869 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:48.652195 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:48.651995 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:49.651060 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:49.651017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:49.651219 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:49.651127 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:50.651912 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:50.651878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:50.652351 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:50.651878 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:50.652351 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:50.652004 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:50.652351 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:50.652095 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:51.651572 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:51.651533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:51.651738 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:51.651649 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:52.294487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:52.294434 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:52.294968 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:52.294538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:52.294968 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.294600 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:52.294968 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.294656 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:52.294968 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.294684 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:07:08.294661901 +0000 UTC m=+34.250116457 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:06:52.294968 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.294711 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:08.294693041 +0000 UTC m=+34.250147606 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:06:52.395668 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:52.395634 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:52.395837 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.395812 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:06:52.395837 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.395834 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:06:52.395837 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.395845 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:52.396034 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.395897 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:08.395882431 +0000 UTC m=+34.351336972 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:06:52.651976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:52.651896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:52.652120 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.652010 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:52.652426 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:52.652399 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:52.652550 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:52.652522 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:53.650977 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:53.650946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:53.651393 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:53.651059 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:54.652203 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:54.652171 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:54.652671 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:54.652272 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:54.652671 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:54.652281 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:54.652671 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:54.652368 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:55.651090 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:55.651054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:55.651235 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:55.651168 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:56.650997 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:56.650964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:56.651483 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:56.650964 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:56.651483 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:56.651091 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:56.651483 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:56.651168 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:57.651482 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.651293 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:57.652061 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:57.651564 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:57.821322 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.821099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" event={"ID":"0dd7a4f18fc1aca27c6a9caeb4dafb15","Type":"ContainerStarted","Data":"f08d5ff9d83b0f8f47fd04bf2de71534583636172801ed3b1dd927cabbb10c57"} Jul 15 13:06:57.822593 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.822570 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-grrjx" event={"ID":"6eead1c0-e669-4f42-be4a-ad182648b50c","Type":"ContainerStarted","Data":"41342f85fb8e0a929b90d7495bf6ff923a70b00acf6d0385a0fed36308283729"} Jul 15 13:06:57.826168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.826147 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-x7jkc" event={"ID":"03957cc5-0ec0-48c6-a682-ab1a0d26a273","Type":"ContainerStarted","Data":"2c8b47c820862a4a2053212edcd514b0edb68f957d548c151fcc2620962898a9"} Jul 15 13:06:57.827448 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.827425 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-c4wgv" event={"ID":"c5b39e97-5dd3-4472-b87f-436050a3656e","Type":"ContainerStarted","Data":"7aed1ae77abed39f6704e44409ceb3bb6921216787d0e965d4dadb731ccca5b5"} Jul 15 13:06:57.828759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.828739 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" event={"ID":"e70c5403-e894-47ff-9b9f-c32be099611a","Type":"ContainerStarted","Data":"1ae3e1141687a24821a06f4b5731098902f091d5a8d670f6c5bf67eba68540a5"} Jul 15 13:06:57.830097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.830079 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nwkk4" event={"ID":"0d4460e2-5dfa-42ab-9921-6a617c2d207d","Type":"ContainerStarted","Data":"a9129e8b13d7f62fa7ecc928a5de6feff552dc5de278c4f014dab1637676eda6"} Jul 15 13:06:57.832335 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.832318 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:06:57.832670 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.832652 2564 generic.go:356] "Generic (PLEG): container finished" podID="2c444ed9-604e-4fae-8437-7b0548983837" containerID="c7461f6c8086b584368dcadd8a089f1a29c118bfabf534295f2c620bb3763be5" exitCode=1 Jul 15 13:06:57.832735 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.832714 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"bd13615819b7b9a19a60ca305a45f10746a332c35667299141969b3463fcfa9b"} Jul 15 13:06:57.832783 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.832765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"7089e446bb140ec1b75a698e6cc4521052fa70ba0993aa0f7903b457c92cfc12"} Jul 15 13:06:57.832822 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.832801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerDied","Data":"c7461f6c8086b584368dcadd8a089f1a29c118bfabf534295f2c620bb3763be5"} Jul 15 13:06:57.832858 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.832836 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"c006aaacc5de2515f6e7e9f247812e6885a5a4f86b7370779432c809621bde57"} Jul 15 13:06:57.834010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.833990 2564 generic.go:356] "Generic (PLEG): container finished" podID="6349c6b9-5951-4fa4-97b5-6d0497b68807" containerID="1a2306bc77df3926d276c7db832723738ff0d34d1e7c453b5ca5241be69e269e" exitCode=0 Jul 15 13:06:57.834093 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.834071 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerDied","Data":"1a2306bc77df3926d276c7db832723738ff0d34d1e7c453b5ca5241be69e269e"} Jul 15 13:06:57.835462 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.835375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jgwwr" event={"ID":"33145dcd-966e-41d4-b9e7-b0c1dfe16528","Type":"ContainerStarted","Data":"779999f929af6987dcb293113a3ca79b1f2491ccc71ee8cbec1b4291feb53aa2"} Jul 15 13:06:57.841141 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.841097 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-149.ec2.internal" podStartSLOduration=21.841086643 podStartE2EDuration="21.841086643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:57.840519561 +0000 UTC m=+23.795974127" watchObservedRunningTime="2026-07-15 13:06:57.841086643 +0000 UTC m=+23.796541206" Jul 15 13:06:57.885601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.885554 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-c4wgv" podStartSLOduration=2.9302631679999998 podStartE2EDuration="22.885535413s" totalImagesPullingTime="19.955272245s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:57.885497525 +0000 UTC m=+23.840952084" watchObservedRunningTime="2026-07-15 13:06:57.885535413 +0000 UTC m=+23.840989977" Jul 15 13:06:57.907672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.907626 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jgwwr" podStartSLOduration=8.384901253 podStartE2EDuration="22.907614182s" totalImagesPullingTime="14.522712929s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:57.906805577 +0000 UTC m=+23.862260139" watchObservedRunningTime="2026-07-15 13:06:57.907614182 +0000 UTC m=+23.863068744" Jul 15 13:06:57.948705 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.948667 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-grrjx" podStartSLOduration=2.706360898 podStartE2EDuration="22.948653166s" totalImagesPullingTime="20.242292268s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:57.948355434 +0000 UTC m=+23.903809997" watchObservedRunningTime="2026-07-15 13:06:57.948653166 +0000 UTC m=+23.904107729" Jul 15 13:06:57.948842 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.948820 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-x7jkc" podStartSLOduration=2.695742951 podStartE2EDuration="22.948811318s" totalImagesPullingTime="20.253068367s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:57.928566919 +0000 UTC m=+23.884021486" watchObservedRunningTime="2026-07-15 13:06:57.948811318 +0000 UTC m=+23.904265882" Jul 15 13:06:57.969777 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:57.969740 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nwkk4" podStartSLOduration=2.7242851359999998 podStartE2EDuration="22.969728121s" totalImagesPullingTime="20.245442985s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:57.969616521 +0000 UTC m=+23.925071084" watchObservedRunningTime="2026-07-15 13:06:57.969728121 +0000 UTC m=+23.925182683" Jul 15 13:06:58.651879 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.651855 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:06:58.652287 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.651856 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:06:58.652287 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:58.651956 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:06:58.652287 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:58.652046 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:06:58.839062 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.839040 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:06:58.839402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.839375 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"1854406f612422792269a215a58fb36ced67d73490d3ef24559f5a473dff4b44"} Jul 15 13:06:58.839552 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.839500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"2b10ac4165c23c3336233e8c490ebb8fb00df77300b997b1198c5f3350e3702e"} Jul 15 13:06:58.840825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.840798 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-b548v" event={"ID":"8a216b80-147e-4f1e-b0cc-a001496017c0","Type":"ContainerStarted","Data":"51cb43272087b2fd681a190cf6cbab40d01f4371cb6ea18ee8a1f2a1a7532542"} Jul 15 13:06:58.855961 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.855910 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-b548v" podStartSLOduration=3.889666437 podStartE2EDuration="23.855892159s" totalImagesPullingTime="19.966225722s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:06:58.855272339 +0000 UTC m=+24.810726902" watchObservedRunningTime="2026-07-15 13:06:58.855892159 +0000 UTC m=+24.811346725" Jul 15 13:06:58.863045 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:58.863024 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 15 13:06:59.510170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:59.510062 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-15T13:06:58.863042381Z","UUID":"2b21d2d2-2fbb-41bf-83ef-c501fe3ad920","Handler":null,"Name":"","Endpoint":""} Jul 15 13:06:59.511752 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:59.511683 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 15 13:06:59.511752 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:59.511726 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 15 13:06:59.651698 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:59.651670 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:06:59.651875 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:06:59.651849 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:06:59.845530 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:06:59.845502 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" event={"ID":"e70c5403-e894-47ff-9b9f-c32be099611a","Type":"ContainerStarted","Data":"f165bfb38e4d775c0c8d89bfb8300786d4360c84c15479e7cbe99bd938560bd2"} Jul 15 13:07:00.651527 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:00.651342 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:00.651730 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:00.651352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:00.651730 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:00.651605 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:07:00.651730 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:00.651680 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:07:00.849571 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:00.849531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" event={"ID":"e70c5403-e894-47ff-9b9f-c32be099611a","Type":"ContainerStarted","Data":"d839128c90f7ab4f28b833cf7f7dd2df8e2e9a7e080eb45615ac58630e83a913"} Jul 15 13:07:00.852667 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:00.852647 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:07:00.853003 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:00.852982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"ee3cae89579114e9ee56bb1e7a969554c337915c2231dcdaa4e627c76cb473d6"} Jul 15 13:07:00.865804 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:00.865766 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8b4lv" podStartSLOduration=3.825435611 podStartE2EDuration="25.865752064s" totalImagesPullingTime="22.040316453s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:00.865744413 +0000 UTC m=+26.821198977" watchObservedRunningTime="2026-07-15 13:07:00.865752064 +0000 UTC m=+26.821206628" Jul 15 13:07:01.651078 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:01.651047 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:01.651260 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:01.651140 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:07:02.463223 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.462980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:07:02.464158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.464137 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:07:02.651538 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.651507 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:02.651698 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:02.651628 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:07:02.651698 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.651677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:02.651805 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:02.651786 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:07:02.861201 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.861131 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:07:02.861610 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.861547 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"c5eb8203ad30508e22e00f515073566bddd2cef642e7905f17f60f693166ae69"} Jul 15 13:07:02.861838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.861815 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:07:02.862073 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.862056 2564 scope.go:117] "RemoveContainer" containerID="c7461f6c8086b584368dcadd8a089f1a29c118bfabf534295f2c620bb3763be5" Jul 15 13:07:02.862202 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:02.862185 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jgwwr" Jul 15 13:07:03.651020 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.650992 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:03.651457 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:03.651098 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:07:03.868798 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.868769 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:07:03.869174 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.869142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" event={"ID":"2c444ed9-604e-4fae-8437-7b0548983837","Type":"ContainerStarted","Data":"9875675f15e2561bf687c33e1da600956a264bc0ba543c46d976bb40e4305844"} Jul 15 13:07:03.869528 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.869504 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:07:03.869690 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.869579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:07:03.869690 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.869629 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:07:03.889054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.887077 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:07:03.890701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.890676 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:07:03.902387 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:03.902306 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" podStartSLOduration=8.651800682 podStartE2EDuration="28.902290536s" totalImagesPullingTime="20.250489854s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:03.901620528 +0000 UTC m=+29.857075094" watchObservedRunningTime="2026-07-15 13:07:03.902290536 +0000 UTC m=+29.857745100" Jul 15 13:07:04.237854 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:04.237787 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgnvn"] Jul 15 13:07:04.237972 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:04.237897 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:04.238005 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:04.237974 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:07:04.240938 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:04.240909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vtkg6"] Jul 15 13:07:04.241054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:04.241025 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:04.241132 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:04.241110 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:07:04.241590 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:04.241572 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5vjbj"] Jul 15 13:07:04.241679 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:04.241667 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:04.241786 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:04.241756 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:07:05.651229 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:05.651199 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:05.651702 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:05.651236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:05.651702 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:05.651344 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:07:05.651702 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:05.651425 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:05.651702 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:05.651423 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:07:05.651702 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:05.651513 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:07:06.878378 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:06.878140 2564 generic.go:356] "Generic (PLEG): container finished" podID="6349c6b9-5951-4fa4-97b5-6d0497b68807" containerID="5e49e876363d8f214e9bf9b89eae5dd3887de1669bbe2f53c74e89a5653a6641" exitCode=0 Jul 15 13:07:06.878378 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:06.878209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerDied","Data":"5e49e876363d8f214e9bf9b89eae5dd3887de1669bbe2f53c74e89a5653a6641"} Jul 15 13:07:07.650865 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:07.650837 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:07.650952 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:07.650870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:07.650952 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:07.650918 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:07.650952 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:07.650928 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:07:07.651113 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:07.651012 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:07:07.651164 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:07.651120 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:07:07.882407 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:07.882322 2564 generic.go:356] "Generic (PLEG): container finished" podID="6349c6b9-5951-4fa4-97b5-6d0497b68807" containerID="ad04882f2c69d90e0797d4948d18a83f51da39a8f2659eb94b46793e40f01ed6" exitCode=0 Jul 15 13:07:07.882407 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:07.882367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerDied","Data":"ad04882f2c69d90e0797d4948d18a83f51da39a8f2659eb94b46793e40f01ed6"} Jul 15 13:07:08.321525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:08.321429 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:08.321670 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.321592 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 15 13:07:08.321670 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.321666 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret podName:18e85af3-fa0a-40e5-9350-2133b1797824 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:40.321646114 +0000 UTC m=+66.277100674 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret") pod "global-pull-secret-syncer-lgnvn" (UID: "18e85af3-fa0a-40e5-9350-2133b1797824") : object "kube-system"/"original-pull-secret" not registered Jul 15 13:07:08.321761 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:08.321716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:08.321843 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.321825 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:07:08.321892 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.321881 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:07:40.321866773 +0000 UTC m=+66.277321319 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 15 13:07:08.422757 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:08.422721 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:08.422917 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.422901 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 15 13:07:08.422963 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.422923 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 15 13:07:08.422963 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.422933 2564 projected.go:196] Error preparing data for projected volume kube-api-access-wpbbl for pod openshift-network-diagnostics/network-check-target-vtkg6: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:07:08.423025 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:08.422991 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl podName:01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1 nodeName:}" failed. No retries permitted until 2026-07-15 13:07:40.422973034 +0000 UTC m=+66.378427595 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-wpbbl" (UniqueName: "kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl") pod "network-check-target-vtkg6" (UID: "01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 15 13:07:08.887176 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:08.887139 2564 generic.go:356] "Generic (PLEG): container finished" podID="6349c6b9-5951-4fa4-97b5-6d0497b68807" containerID="c29496b04c25539aee0f9ee46bf5d390d0ef658cdc579460ff466307c3f039d6" exitCode=0 Jul 15 13:07:08.887558 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:08.887200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerDied","Data":"c29496b04c25539aee0f9ee46bf5d390d0ef658cdc579460ff466307c3f039d6"} Jul 15 13:07:09.651524 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:09.651483 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:09.651693 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:09.651581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:09.651693 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:09.651584 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vtkg6" podUID="01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1" Jul 15 13:07:09.651693 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:09.651590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:09.651839 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:09.651695 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:07:09.651839 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:09.651754 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-lgnvn" podUID="18e85af3-fa0a-40e5-9350-2133b1797824" Jul 15 13:07:10.274739 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.274711 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-149.ec2.internal" event="NodeReady" Jul 15 13:07:10.275236 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.274839 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 15 13:07:10.324316 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.324291 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wvk9l"] Jul 15 13:07:10.326108 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.326094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.328170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.328147 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kgndk"] Jul 15 13:07:10.329126 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.329102 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-txt24\"" Jul 15 13:07:10.329390 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.329373 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 15 13:07:10.329497 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.329403 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 15 13:07:10.329662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.329645 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.331677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.331662 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 15 13:07:10.331956 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.331940 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 15 13:07:10.332086 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.331963 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 15 13:07:10.332298 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.332283 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-htl88\"" Jul 15 13:07:10.337035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.337016 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.337129 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.337081 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a06378-bbbd-412e-95fe-21d586ec36ea-tmp-dir\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.337129 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.337118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb2ps\" (UniqueName: \"kubernetes.io/projected/65a06378-bbbd-412e-95fe-21d586ec36ea-kube-api-access-gb2ps\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.337205 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.337162 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65a06378-bbbd-412e-95fe-21d586ec36ea-config-volume\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.338817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.338798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wvk9l"] Jul 15 13:07:10.341402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.341383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kgndk"] Jul 15 13:07:10.437962 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.437931 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a06378-bbbd-412e-95fe-21d586ec36ea-tmp-dir\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.438101 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.437968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb2ps\" (UniqueName: \"kubernetes.io/projected/65a06378-bbbd-412e-95fe-21d586ec36ea-kube-api-access-gb2ps\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.438101 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.437994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65a06378-bbbd-412e-95fe-21d586ec36ea-config-volume\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.438274 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.438255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0030efc9-4bad-477c-9a8c-4ba925b0e420-cert\") pod \"ingress-canary-kgndk\" (UID: \"0030efc9-4bad-477c-9a8c-4ba925b0e420\") " pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.438312 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.438298 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.438347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.438328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4j6f\" (UniqueName: \"kubernetes.io/projected/0030efc9-4bad-477c-9a8c-4ba925b0e420-kube-api-access-n4j6f\") pod \"ingress-canary-kgndk\" (UID: \"0030efc9-4bad-477c-9a8c-4ba925b0e420\") " pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.438382 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.438365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65a06378-bbbd-412e-95fe-21d586ec36ea-tmp-dir\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.438432 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:10.438417 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:10.438505 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:10.438496 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:07:10.938481304 +0000 UTC m=+36.893935857 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:10.438505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.438492 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/65a06378-bbbd-412e-95fe-21d586ec36ea-config-volume\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.448053 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.448029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb2ps\" (UniqueName: \"kubernetes.io/projected/65a06378-bbbd-412e-95fe-21d586ec36ea-kube-api-access-gb2ps\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.539185 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.539103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0030efc9-4bad-477c-9a8c-4ba925b0e420-cert\") pod \"ingress-canary-kgndk\" (UID: \"0030efc9-4bad-477c-9a8c-4ba925b0e420\") " pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.539185 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.539148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4j6f\" (UniqueName: \"kubernetes.io/projected/0030efc9-4bad-477c-9a8c-4ba925b0e420-kube-api-access-n4j6f\") pod \"ingress-canary-kgndk\" (UID: \"0030efc9-4bad-477c-9a8c-4ba925b0e420\") " pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.541556 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.541538 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0030efc9-4bad-477c-9a8c-4ba925b0e420-cert\") pod \"ingress-canary-kgndk\" (UID: \"0030efc9-4bad-477c-9a8c-4ba925b0e420\") " pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.547546 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.547528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4j6f\" (UniqueName: \"kubernetes.io/projected/0030efc9-4bad-477c-9a8c-4ba925b0e420-kube-api-access-n4j6f\") pod \"ingress-canary-kgndk\" (UID: \"0030efc9-4bad-477c-9a8c-4ba925b0e420\") " pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.642842 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.642809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kgndk" Jul 15 13:07:10.774943 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.774765 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kgndk"] Jul 15 13:07:10.777825 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:07:10.777800 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0030efc9_4bad_477c_9a8c_4ba925b0e420.slice/crio-10eea4a49bb8255767a9cdc86f8d15ba92ac3244ee6d092e03b85726fbde65bf WatchSource:0}: Error finding container 10eea4a49bb8255767a9cdc86f8d15ba92ac3244ee6d092e03b85726fbde65bf: Status 404 returned error can't find the container with id 10eea4a49bb8255767a9cdc86f8d15ba92ac3244ee6d092e03b85726fbde65bf Jul 15 13:07:10.892015 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.891920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kgndk" event={"ID":"0030efc9-4bad-477c-9a8c-4ba925b0e420","Type":"ContainerStarted","Data":"10eea4a49bb8255767a9cdc86f8d15ba92ac3244ee6d092e03b85726fbde65bf"} Jul 15 13:07:10.942569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:10.942544 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:10.942692 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:10.942662 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:10.942730 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:10.942713 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:07:11.942699174 +0000 UTC m=+37.898153715 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:11.651520 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.651478 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:11.651520 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.651514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:11.652025 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.651649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:11.654689 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.654669 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 13:07:11.655801 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.655783 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 13:07:11.655894 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.655825 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 13:07:11.655894 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.655856 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8tgt2\"" Jul 15 13:07:11.655999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.655969 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 13:07:11.656072 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.656055 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nr6v2\"" Jul 15 13:07:11.950435 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:11.950346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:11.950665 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:11.950509 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:11.950665 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:11.950588 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:07:13.950566614 +0000 UTC m=+39.906021175 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:13.901819 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:13.901584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kgndk" event={"ID":"0030efc9-4bad-477c-9a8c-4ba925b0e420","Type":"ContainerStarted","Data":"840f61c5c7f4209f16b0df841186e3428176886e70013aaef0262208af46d49c"} Jul 15 13:07:13.921058 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:13.920999 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kgndk" podStartSLOduration=1.541923432 podStartE2EDuration="3.920978238s" totalImagesPullingTime="2.379054806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:13.920531877 +0000 UTC m=+39.875986476" watchObservedRunningTime="2026-07-15 13:07:13.920978238 +0000 UTC m=+39.876432806" Jul 15 13:07:13.963962 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:13.963932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:13.964124 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:13.964078 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:13.964195 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:13.964151 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:07:17.964130193 +0000 UTC m=+43.919584754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:15.908655 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:15.908624 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerStarted","Data":"2ecedc97167f78ffe80ce96bb3c683467b6993f1a3d62b96bfe7771379c611dc"} Jul 15 13:07:16.912816 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:16.912779 2564 generic.go:356] "Generic (PLEG): container finished" podID="6349c6b9-5951-4fa4-97b5-6d0497b68807" containerID="2ecedc97167f78ffe80ce96bb3c683467b6993f1a3d62b96bfe7771379c611dc" exitCode=0 Jul 15 13:07:16.913186 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:16.912833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerDied","Data":"2ecedc97167f78ffe80ce96bb3c683467b6993f1a3d62b96bfe7771379c611dc"} Jul 15 13:07:17.918035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:17.918001 2564 generic.go:356] "Generic (PLEG): container finished" podID="6349c6b9-5951-4fa4-97b5-6d0497b68807" containerID="4800972e8ab3a1b7ba9e3611ed840c4c749931391f35ae29b792b3166c2b4d80" exitCode=0 Jul 15 13:07:17.918398 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:17.918055 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerDied","Data":"4800972e8ab3a1b7ba9e3611ed840c4c749931391f35ae29b792b3166c2b4d80"} Jul 15 13:07:17.992360 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:17.992330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:17.992456 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:17.992409 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:17.992523 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:17.992486 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:07:25.992452093 +0000 UTC m=+51.947906634 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:18.925685 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:18.925652 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-spqmk" event={"ID":"6349c6b9-5951-4fa4-97b5-6d0497b68807","Type":"ContainerStarted","Data":"77e30d98f2ffbeeefdededcf558a51562c606849f9e656e2d1886ade68a56a2a"} Jul 15 13:07:18.951162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:18.951113 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-spqmk" podStartSLOduration=7.327644849 podStartE2EDuration="43.951094941s" totalImagesPullingTime="36.623450092s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:18.949747659 +0000 UTC m=+44.905202222" watchObservedRunningTime="2026-07-15 13:07:18.951094941 +0000 UTC m=+44.906549505" Jul 15 13:07:26.045181 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:26.045137 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:26.045611 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:26.045252 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:26.045611 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:26.045329 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:07:42.045311841 +0000 UTC m=+68.000766386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:35.888196 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:35.888168 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-ghhfs" Jul 15 13:07:40.343059 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.343014 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:07:40.343450 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.343074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:40.345801 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.345778 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 15 13:07:40.345938 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.345917 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 15 13:07:40.353919 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:40.353896 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 13:07:40.353984 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:40.353970 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:08:44.353954209 +0000 UTC m=+130.309408764 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : secret "metrics-daemon-secret" not found Jul 15 13:07:40.357623 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.357604 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/18e85af3-fa0a-40e5-9350-2133b1797824-original-pull-secret\") pod \"global-pull-secret-syncer-lgnvn\" (UID: \"18e85af3-fa0a-40e5-9350-2133b1797824\") " pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:40.444316 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.444271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:40.446601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.446580 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 15 13:07:40.457709 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.457688 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 15 13:07:40.463588 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.463571 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-lgnvn" Jul 15 13:07:40.467925 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.467903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbbl\" (UniqueName: \"kubernetes.io/projected/01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1-kube-api-access-wpbbl\") pod \"network-check-target-vtkg6\" (UID: \"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1\") " pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:40.473930 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.473911 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nr6v2\"" Jul 15 13:07:40.481995 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.481972 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:40.614328 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.614099 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-lgnvn"] Jul 15 13:07:40.617966 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:07:40.617940 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e85af3_fa0a_40e5_9350_2133b1797824.slice/crio-5681aa8b267cecf56a8fbe7ab9a4d523a6dd0bfff8c20f5f4489194b3e214c10 WatchSource:0}: Error finding container 5681aa8b267cecf56a8fbe7ab9a4d523a6dd0bfff8c20f5f4489194b3e214c10: Status 404 returned error can't find the container with id 5681aa8b267cecf56a8fbe7ab9a4d523a6dd0bfff8c20f5f4489194b3e214c10 Jul 15 13:07:40.627662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.627628 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vtkg6"] Jul 15 13:07:40.629847 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:07:40.629827 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01f0ecdb_e90b_401d_a4ea_fcc9f1c60bc1.slice/crio-45bf1a2110774ecfa1404201f08b54bbf0df523e351a488da81d85d643d9873c WatchSource:0}: Error finding container 45bf1a2110774ecfa1404201f08b54bbf0df523e351a488da81d85d643d9873c: Status 404 returned error can't find the container with id 45bf1a2110774ecfa1404201f08b54bbf0df523e351a488da81d85d643d9873c Jul 15 13:07:40.977799 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.977724 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vtkg6" event={"ID":"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1","Type":"ContainerStarted","Data":"45bf1a2110774ecfa1404201f08b54bbf0df523e351a488da81d85d643d9873c"} Jul 15 13:07:40.979079 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:40.979051 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgnvn" event={"ID":"18e85af3-fa0a-40e5-9350-2133b1797824","Type":"ContainerStarted","Data":"5681aa8b267cecf56a8fbe7ab9a4d523a6dd0bfff8c20f5f4489194b3e214c10"} Jul 15 13:07:42.056159 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:42.055970 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:07:42.056703 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:42.056221 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:07:42.056703 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:07:42.056325 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:08:14.056301912 +0000 UTC m=+100.011756467 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:07:44.991920 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:44.991882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vtkg6" event={"ID":"01f0ecdb-e90b-401d-a4ea-fcc9f1c60bc1","Type":"ContainerStarted","Data":"8abfaea92ceeadd34bafbd0bbfa5066a82a50f2477623e45a8e81b11c19fa697"} Jul 15 13:07:44.992391 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:44.991978 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:07:44.993373 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:44.993345 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-lgnvn" event={"ID":"18e85af3-fa0a-40e5-9350-2133b1797824","Type":"ContainerStarted","Data":"37ca221af180f737c9351fd37d3f9213a0d92ea2a318daf1638ec3b4b88384f1"} Jul 15 13:07:45.009938 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:45.009877 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vtkg6" podStartSLOduration=65.818571633 podStartE2EDuration="1m10.009861035s" totalImagesPullingTime="4.191289402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:45.009002679 +0000 UTC m=+70.964457242" watchObservedRunningTime="2026-07-15 13:07:45.009861035 +0000 UTC m=+70.965315598" Jul 15 13:07:45.024139 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:07:45.024083 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-lgnvn" podStartSLOduration=65.814899638 podStartE2EDuration="1m10.024065352s" totalImagesPullingTime="4.209165714s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:07:45.024008781 +0000 UTC m=+70.979463345" watchObservedRunningTime="2026-07-15 13:07:45.024065352 +0000 UTC m=+70.979519912" Jul 15 13:08:14.077077 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:08:14.076994 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:08:14.077500 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:08:14.077113 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:08:14.077500 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:08:14.077165 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:09:18.077150945 +0000 UTC m=+164.032605486 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:08:15.998327 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:08:15.998298 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vtkg6" Jul 15 13:08:44.382341 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:08:44.382303 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:08:44.382830 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:08:44.382451 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 15 13:08:44.382830 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:08:44.382536 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs podName:dea320b5-7fcb-48e0-bf9e-b9475cfb141a nodeName:}" failed. No retries permitted until 2026-07-15 13:10:46.382519131 +0000 UTC m=+252.337973672 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs") pod "network-metrics-daemon-5vjbj" (UID: "dea320b5-7fcb-48e0-bf9e-b9475cfb141a") : secret "metrics-daemon-secret" not found Jul 15 13:09:13.336630 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:09:13.336587 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wvk9l" podUID="65a06378-bbbd-412e-95fe-21d586ec36ea" Jul 15 13:09:14.206779 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:09:14.206699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvk9l" Jul 15 13:09:14.678528 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:09:14.678464 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-5vjbj" podUID="dea320b5-7fcb-48e0-bf9e-b9475cfb141a" Jul 15 13:09:18.104130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:09:18.104095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:09:18.104494 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:09:18.104244 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 15 13:09:18.104494 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:09:18.104316 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls podName:65a06378-bbbd-412e-95fe-21d586ec36ea nodeName:}" failed. No retries permitted until 2026-07-15 13:11:20.10429972 +0000 UTC m=+286.059754264 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls") pod "dns-default-wvk9l" (UID: "65a06378-bbbd-412e-95fe-21d586ec36ea") : secret "dns-default-metrics-tls" not found Jul 15 13:09:27.651632 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:09:27.651541 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:10:27.177533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.177498 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb"] Jul 15 13:10:27.179547 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.179527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" Jul 15 13:10:27.181873 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.181854 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:10:27.182024 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.182003 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-657cw\"" Jul 15 13:10:27.182700 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.182687 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 15 13:10:27.187415 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.187392 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw"] Jul 15 13:10:27.189161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.189146 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66dd8dbb64-8vkwx"] Jul 15 13:10:27.189271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.189258 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.190961 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.190941 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.191317 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.191297 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 15 13:10:27.191421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.191378 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:10:27.191498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.191428 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mjh54\"" Jul 15 13:10:27.191563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.191531 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 15 13:10:27.191563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.191533 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 15 13:10:27.194089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194072 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 15 13:10:27.194299 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194282 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 15 13:10:27.194389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194298 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 15 13:10:27.194389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194352 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 15 13:10:27.194389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194299 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 15 13:10:27.194736 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194715 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 15 13:10:27.194858 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.194839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-25vfg\"" Jul 15 13:10:27.195504 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.195486 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb"] Jul 15 13:10:27.234770 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.234745 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw"] Jul 15 13:10:27.251304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251284 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-default-certificate\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.251420 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-stats-auth\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.251420 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251328 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.251420 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251352 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c783ddea-55cf-4263-8db7-d8d43cea99f2-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.251420 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251406 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.251611 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2zvd\" (UniqueName: \"kubernetes.io/projected/8dbe8e4b-d986-49e1-a374-01d806777597-kube-api-access-x2zvd\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.251611 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb69w\" (UniqueName: \"kubernetes.io/projected/dd675c67-109e-455e-a40b-97166836645a-kube-api-access-qb69w\") pod \"volume-data-source-validator-78495597f6-84dnb\" (UID: \"dd675c67-109e-455e-a40b-97166836645a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" Jul 15 13:10:27.251611 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251496 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29wvf\" (UniqueName: \"kubernetes.io/projected/c783ddea-55cf-4263-8db7-d8d43cea99f2-kube-api-access-29wvf\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.251611 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.251529 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c783ddea-55cf-4263-8db7-d8d43cea99f2-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.258223 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.258198 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66dd8dbb64-8vkwx"] Jul 15 13:10:27.287325 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.287296 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp"] Jul 15 13:10:27.289335 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.289316 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.291699 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.291678 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 15 13:10:27.291791 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.291736 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:10:27.292138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.292122 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7dhmn\"" Jul 15 13:10:27.292253 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.292238 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 15 13:10:27.308445 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.308424 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp"] Jul 15 13:10:27.351796 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c783ddea-55cf-4263-8db7-d8d43cea99f2-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.351932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-default-certificate\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.351932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-stats-auth\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.351932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.351932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351859 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c783ddea-55cf-4263-8db7-d8d43cea99f2-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.351932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6tm7\" (UniqueName: \"kubernetes.io/projected/f6ecc040-d4fc-4de9-90ba-660e77e32b54-kube-api-access-z6tm7\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.351932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351932 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.352237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.351960 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.352237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.351961 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 13:10:27.352237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.352107 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2zvd\" (UniqueName: \"kubernetes.io/projected/8dbe8e4b-d986-49e1-a374-01d806777597-kube-api-access-x2zvd\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.352237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.352152 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:27.852114896 +0000 UTC m=+233.807569442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : secret "router-metrics-certs-default" not found Jul 15 13:10:27.352237 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.352229 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:27.852216185 +0000 UTC m=+233.807670769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : configmap references non-existent config key: service-ca.crt Jul 15 13:10:27.352669 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.352247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb69w\" (UniqueName: \"kubernetes.io/projected/dd675c67-109e-455e-a40b-97166836645a-kube-api-access-qb69w\") pod \"volume-data-source-validator-78495597f6-84dnb\" (UID: \"dd675c67-109e-455e-a40b-97166836645a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" Jul 15 13:10:27.352669 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.352314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29wvf\" (UniqueName: \"kubernetes.io/projected/c783ddea-55cf-4263-8db7-d8d43cea99f2-kube-api-access-29wvf\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.352669 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.352439 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c783ddea-55cf-4263-8db7-d8d43cea99f2-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.354216 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.354193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c783ddea-55cf-4263-8db7-d8d43cea99f2-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.354355 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.354340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-stats-auth\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.354395 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.354343 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-default-certificate\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.367681 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.367657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2zvd\" (UniqueName: \"kubernetes.io/projected/8dbe8e4b-d986-49e1-a374-01d806777597-kube-api-access-x2zvd\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.369237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.369212 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb69w\" (UniqueName: \"kubernetes.io/projected/dd675c67-109e-455e-a40b-97166836645a-kube-api-access-qb69w\") pod \"volume-data-source-validator-78495597f6-84dnb\" (UID: \"dd675c67-109e-455e-a40b-97166836645a\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" Jul 15 13:10:27.369429 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.369410 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-29wvf\" (UniqueName: \"kubernetes.io/projected/c783ddea-55cf-4263-8db7-d8d43cea99f2-kube-api-access-29wvf\") pod \"kube-storage-version-migrator-operator-799cb7f958-2fjtw\" (UID: \"c783ddea-55cf-4263-8db7-d8d43cea99f2\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.384378 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.384358 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8698fd8df6-jzrgw"] Jul 15 13:10:27.386881 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.386866 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.392175 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.392154 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 15 13:10:27.392273 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.392158 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 15 13:10:27.392273 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.392256 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j4np7\"" Jul 15 13:10:27.392361 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.392335 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 15 13:10:27.398341 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.398320 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 15 13:10:27.410534 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.410510 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8698fd8df6-jzrgw"] Jul 15 13:10:27.453379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453309 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-image-registry-private-configuration\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453374 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-certificates\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78qrt\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-kube-api-access-78qrt\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453495 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-installation-pull-secrets\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6tm7\" (UniqueName: \"kubernetes.io/projected/f6ecc040-d4fc-4de9-90ba-660e77e32b54-kube-api-access-z6tm7\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.453729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453583 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-ca-trust-extracted\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.453729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-trusted-ca\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.453713 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-bound-sa-token\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.453729 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.453721 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 13:10:27.453970 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.453776 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls podName:f6ecc040-d4fc-4de9-90ba-660e77e32b54 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:27.95375641 +0000 UTC m=+233.909210952 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls") pod "cluster-samples-operator-65db858894-2qsqp" (UID: "f6ecc040-d4fc-4de9-90ba-660e77e32b54") : secret "samples-operator-tls" not found Jul 15 13:10:27.465901 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.465873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6tm7\" (UniqueName: \"kubernetes.io/projected/f6ecc040-d4fc-4de9-90ba-660e77e32b54-kube-api-access-z6tm7\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.488539 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.488516 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" Jul 15 13:10:27.499374 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.499357 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" Jul 15 13:10:27.554253 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554095 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-ca-trust-extracted\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554253 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554170 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-trusted-ca\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554253 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-bound-sa-token\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554253 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-image-registry-private-configuration\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554280 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-certificates\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554331 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78qrt\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-kube-api-access-78qrt\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.554368 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-installation-pull-secrets\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.554786 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.554595 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:10:27.554786 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.554616 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698fd8df6-jzrgw: secret "image-registry-tls" not found Jul 15 13:10:27.555225 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.554925 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls podName:4c0ef71e-cd6b-4c2f-96b3-92d22906c837 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:28.054791595 +0000 UTC m=+234.010246156 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls") pod "image-registry-8698fd8df6-jzrgw" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837") : secret "image-registry-tls" not found Jul 15 13:10:27.560830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.556527 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-ca-trust-extracted\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.560830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.557083 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-trusted-ca\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.560830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.557135 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-certificates\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.560830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.557540 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-installation-pull-secrets\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.560830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.558898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-image-registry-private-configuration\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.564865 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.564844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-bound-sa-token\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.566812 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.566781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78qrt\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-kube-api-access-78qrt\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:27.618113 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.618083 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb"] Jul 15 13:10:27.620912 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:10:27.620887 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd675c67_109e_455e_a40b_97166836645a.slice/crio-aa14e236c703a12b5a843236cc99dcfb521cd0567234c22103654cc712fb4e62 WatchSource:0}: Error finding container aa14e236c703a12b5a843236cc99dcfb521cd0567234c22103654cc712fb4e62: Status 404 returned error can't find the container with id aa14e236c703a12b5a843236cc99dcfb521cd0567234c22103654cc712fb4e62 Jul 15 13:10:27.633207 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.633184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw"] Jul 15 13:10:27.635679 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:10:27.635656 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc783ddea_55cf_4263_8db7_d8d43cea99f2.slice/crio-6715e4e5f8cde94ca9953def8cbee1cfcaee41f033fdf561d650325fbab5efa6 WatchSource:0}: Error finding container 6715e4e5f8cde94ca9953def8cbee1cfcaee41f033fdf561d650325fbab5efa6: Status 404 returned error can't find the container with id 6715e4e5f8cde94ca9953def8cbee1cfcaee41f033fdf561d650325fbab5efa6 Jul 15 13:10:27.856035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.856000 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.856187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.856043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:27.856187 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.856149 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 13:10:27.856263 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.856161 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:28.856143987 +0000 UTC m=+234.811598530 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : configmap references non-existent config key: service-ca.crt Jul 15 13:10:27.856263 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.856220 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:28.856206254 +0000 UTC m=+234.811660797 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : secret "router-metrics-certs-default" not found Jul 15 13:10:27.956367 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:27.956313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:27.956552 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.956459 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 13:10:27.956552 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:27.956537 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls podName:f6ecc040-d4fc-4de9-90ba-660e77e32b54 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:28.956521965 +0000 UTC m=+234.911976507 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls") pod "cluster-samples-operator-65db858894-2qsqp" (UID: "f6ecc040-d4fc-4de9-90ba-660e77e32b54") : secret "samples-operator-tls" not found Jul 15 13:10:28.057026 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:28.056976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:28.057204 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.057150 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:10:28.057204 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.057172 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698fd8df6-jzrgw: secret "image-registry-tls" not found Jul 15 13:10:28.057313 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.057235 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls podName:4c0ef71e-cd6b-4c2f-96b3-92d22906c837 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:29.057215586 +0000 UTC m=+235.012670148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls") pod "image-registry-8698fd8df6-jzrgw" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837") : secret "image-registry-tls" not found Jul 15 13:10:28.386038 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:28.386004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" event={"ID":"dd675c67-109e-455e-a40b-97166836645a","Type":"ContainerStarted","Data":"aa14e236c703a12b5a843236cc99dcfb521cd0567234c22103654cc712fb4e62"} Jul 15 13:10:28.387152 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:28.387122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" event={"ID":"c783ddea-55cf-4263-8db7-d8d43cea99f2","Type":"ContainerStarted","Data":"6715e4e5f8cde94ca9953def8cbee1cfcaee41f033fdf561d650325fbab5efa6"} Jul 15 13:10:28.863657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:28.863620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:28.863825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:28.863707 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:28.863825 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.863761 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:30.86374372 +0000 UTC m=+236.819198264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : configmap references non-existent config key: service-ca.crt Jul 15 13:10:28.863825 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.863808 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 13:10:28.863941 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.863846 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:30.863835497 +0000 UTC m=+236.819290039 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : secret "router-metrics-certs-default" not found Jul 15 13:10:28.964525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:28.964433 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:28.964677 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.964599 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 13:10:28.964677 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:28.964670 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls podName:f6ecc040-d4fc-4de9-90ba-660e77e32b54 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:30.964652237 +0000 UTC m=+236.920106794 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls") pod "cluster-samples-operator-65db858894-2qsqp" (UID: "f6ecc040-d4fc-4de9-90ba-660e77e32b54") : secret "samples-operator-tls" not found Jul 15 13:10:29.065849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:29.065815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:29.066042 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:29.065975 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:10:29.066042 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:29.065993 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698fd8df6-jzrgw: secret "image-registry-tls" not found Jul 15 13:10:29.066158 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:29.066045 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls podName:4c0ef71e-cd6b-4c2f-96b3-92d22906c837 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:31.066029563 +0000 UTC m=+237.021484104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls") pod "image-registry-8698fd8df6-jzrgw" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837") : secret "image-registry-tls" not found Jul 15 13:10:29.391733 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:29.391690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" event={"ID":"dd675c67-109e-455e-a40b-97166836645a","Type":"ContainerStarted","Data":"6714e6169018c54d3fa4719105311f06e9df0852434eaaecf151e55502a83918"} Jul 15 13:10:29.411303 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:29.411251 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-84dnb" podStartSLOduration=1.318670898 podStartE2EDuration="2.411234255s" totalImagesPullingTime="1.092563357s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:10:29.410571403 +0000 UTC m=+235.366025966" watchObservedRunningTime="2026-07-15 13:10:29.411234255 +0000 UTC m=+235.366688850" Jul 15 13:10:30.395995 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:30.395955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" event={"ID":"c783ddea-55cf-4263-8db7-d8d43cea99f2","Type":"ContainerStarted","Data":"f0d103adb7f4c32a49857604cdba1b866aa6b4c0a71973720a13e6c14ca13c87"} Jul 15 13:10:30.414964 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:30.414916 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" podStartSLOduration=1.524939458 podStartE2EDuration="3.414899724s" totalImagesPullingTime="1.889960266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:10:30.414109639 +0000 UTC m=+236.369564210" watchObservedRunningTime="2026-07-15 13:10:30.414899724 +0000 UTC m=+236.370354289" Jul 15 13:10:30.879606 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:30.879563 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:30.879877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:30.879655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:30.879877 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:30.879753 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 13:10:30.879877 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:30.879768 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:34.87974623 +0000 UTC m=+240.835200789 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : configmap references non-existent config key: service-ca.crt Jul 15 13:10:30.879877 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:30.879797 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:34.879785914 +0000 UTC m=+240.835240457 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : secret "router-metrics-certs-default" not found Jul 15 13:10:30.980825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:30.980787 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:30.980977 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:30.980929 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 13:10:30.981018 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:30.980989 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls podName:f6ecc040-d4fc-4de9-90ba-660e77e32b54 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:34.980972711 +0000 UTC m=+240.936427252 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls") pod "cluster-samples-operator-65db858894-2qsqp" (UID: "f6ecc040-d4fc-4de9-90ba-660e77e32b54") : secret "samples-operator-tls" not found Jul 15 13:10:31.081926 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.081894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:31.082063 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.082050 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:10:31.082101 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.082069 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698fd8df6-jzrgw: secret "image-registry-tls" not found Jul 15 13:10:31.082134 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.082124 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls podName:4c0ef71e-cd6b-4c2f-96b3-92d22906c837 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:35.082106366 +0000 UTC m=+241.037561111 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls") pod "image-registry-8698fd8df6-jzrgw" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837") : secret "image-registry-tls" not found Jul 15 13:10:31.830336 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.830301 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8hqsl"] Jul 15 13:10:31.832765 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.832747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.837010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.836987 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 15 13:10:31.837303 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.837281 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 15 13:10:31.837303 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.837285 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9psk\"" Jul 15 13:10:31.837509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.837289 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 15 13:10:31.837632 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.837618 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 15 13:10:31.846332 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.846311 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8hqsl"] Jul 15 13:10:31.888758 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.888731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.888869 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.888780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/499499f3-41c2-4699-98d0-69c986e2faf3-data-volume\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.888869 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.888811 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.888869 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.888846 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76h5m\" (UniqueName: \"kubernetes.io/projected/499499f3-41c2-4699-98d0-69c986e2faf3-kube-api-access-76h5m\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.888869 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.888865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/499499f3-41c2-4699-98d0-69c986e2faf3-crio-socket\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989346 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989312 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989511 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989379 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/499499f3-41c2-4699-98d0-69c986e2faf3-data-volume\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989511 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989417 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989511 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.989446 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:31.989511 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989455 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76h5m\" (UniqueName: \"kubernetes.io/projected/499499f3-41c2-4699-98d0-69c986e2faf3-kube-api-access-76h5m\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989707 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989514 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/499499f3-41c2-4699-98d0-69c986e2faf3-crio-socket\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989707 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.989536 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:32.489514505 +0000 UTC m=+238.444969065 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : configmap "kube-rbac-proxy" not found Jul 15 13:10:31.989707 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.989570 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:31.989707 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989575 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/499499f3-41c2-4699-98d0-69c986e2faf3-crio-socket\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.989707 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:31.989626 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:32.489609292 +0000 UTC m=+238.445063845 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:31.989942 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.989757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/499499f3-41c2-4699-98d0-69c986e2faf3-data-volume\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:31.997806 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:31.997775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-76h5m\" (UniqueName: \"kubernetes.io/projected/499499f3-41c2-4699-98d0-69c986e2faf3-kube-api-access-76h5m\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:32.494802 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:32.494747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:32.494997 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:32.494863 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:32.494997 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:32.494876 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:32.494997 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:32.494937 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:33.494920923 +0000 UTC m=+239.450375463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : configmap "kube-rbac-proxy" not found Jul 15 13:10:32.494997 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:32.494986 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:32.495223 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:32.495041 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:33.495025637 +0000 UTC m=+239.450480197 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:33.503464 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:33.503430 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:33.504028 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:33.503515 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:33.504028 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:33.503575 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:33.504028 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:33.503662 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:35.50363949 +0000 UTC m=+241.459094056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : configmap "kube-rbac-proxy" not found Jul 15 13:10:33.504028 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:33.503664 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:33.504028 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:33.503705 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:35.50369384 +0000 UTC m=+241.459148381 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:34.412128 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:34.412104 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c4wgv_c5b39e97-5dd3-4472-b87f-436050a3656e/dns-node-resolver/0.log" Jul 15 13:10:34.914538 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:34.914503 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:34.914985 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:34.914578 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:34.914985 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:34.914673 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 13:10:34.914985 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:34.914689 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:42.914675449 +0000 UTC m=+248.870129990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : configmap references non-existent config key: service-ca.crt Jul 15 13:10:34.914985 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:34.914736 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:42.914719382 +0000 UTC m=+248.870173924 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : secret "router-metrics-certs-default" not found Jul 15 13:10:35.015531 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.015493 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:35.015702 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.015632 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 15 13:10:35.015702 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.015697 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls podName:f6ecc040-d4fc-4de9-90ba-660e77e32b54 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:43.015682776 +0000 UTC m=+248.971137320 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls") pod "cluster-samples-operator-65db858894-2qsqp" (UID: "f6ecc040-d4fc-4de9-90ba-660e77e32b54") : secret "samples-operator-tls" not found Jul 15 13:10:35.044683 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.044645 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-fh6zv"] Jul 15 13:10:35.047144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.047129 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.049553 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.049529 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 15 13:10:35.049682 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.049552 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 15 13:10:35.050358 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.050342 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 15 13:10:35.050443 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.050358 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 15 13:10:35.050443 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.050401 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ggwll\"" Jul 15 13:10:35.057733 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.057713 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-fh6zv"] Jul 15 13:10:35.115905 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.115870 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88d80b94-5bd0-414c-ba80-343054aae400-signing-cabundle\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.115905 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.115905 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4qn\" (UniqueName: \"kubernetes.io/projected/88d80b94-5bd0-414c-ba80-343054aae400-kube-api-access-xs4qn\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.116094 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.115933 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:35.116094 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.116031 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 15 13:10:35.116094 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.116047 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8698fd8df6-jzrgw: secret "image-registry-tls" not found Jul 15 13:10:35.116094 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.116083 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88d80b94-5bd0-414c-ba80-343054aae400-signing-key\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.116212 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.116097 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls podName:4c0ef71e-cd6b-4c2f-96b3-92d22906c837 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:43.116080897 +0000 UTC m=+249.071535442 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls") pod "image-registry-8698fd8df6-jzrgw" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837") : secret "image-registry-tls" not found Jul 15 13:10:35.217532 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.217444 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88d80b94-5bd0-414c-ba80-343054aae400-signing-key\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.217658 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.217632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88d80b94-5bd0-414c-ba80-343054aae400-signing-cabundle\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.217658 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.217655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4qn\" (UniqueName: \"kubernetes.io/projected/88d80b94-5bd0-414c-ba80-343054aae400-kube-api-access-xs4qn\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.218327 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.218302 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/88d80b94-5bd0-414c-ba80-343054aae400-signing-cabundle\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.219713 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.219694 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/88d80b94-5bd0-414c-ba80-343054aae400-signing-key\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.226329 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.226302 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4qn\" (UniqueName: \"kubernetes.io/projected/88d80b94-5bd0-414c-ba80-343054aae400-kube-api-access-xs4qn\") pod \"service-ca-79f68b9f4f-fh6zv\" (UID: \"88d80b94-5bd0-414c-ba80-343054aae400\") " pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.355202 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.355151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" Jul 15 13:10:35.475548 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.475453 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-fh6zv"] Jul 15 13:10:35.478144 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:10:35.478105 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d80b94_5bd0_414c_ba80_343054aae400.slice/crio-8c9beae4ca5db097fab5b72c25d99872f45656720e3062e5d0ee94b4da817378 WatchSource:0}: Error finding container 8c9beae4ca5db097fab5b72c25d99872f45656720e3062e5d0ee94b4da817378: Status 404 returned error can't find the container with id 8c9beae4ca5db097fab5b72c25d99872f45656720e3062e5d0ee94b4da817378 Jul 15 13:10:35.521278 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.521253 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:35.521401 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.521324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:35.521401 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.521397 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:35.521503 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.521401 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:35.521503 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.521442 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:39.521427654 +0000 UTC m=+245.476882194 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : configmap "kube-rbac-proxy" not found Jul 15 13:10:35.521503 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:35.521457 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:39.521448387 +0000 UTC m=+245.476902927 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:35.812893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:35.812865 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x7jkc_03957cc5-0ec0-48c6-a682-ab1a0d26a273/node-ca/0.log" Jul 15 13:10:36.412634 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:36.412598 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kgndk_0030efc9-4bad-477c-9a8c-4ba925b0e420/serve-healthcheck-canary/0.log" Jul 15 13:10:36.415729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:36.415692 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" event={"ID":"88d80b94-5bd0-414c-ba80-343054aae400","Type":"ContainerStarted","Data":"8c9beae4ca5db097fab5b72c25d99872f45656720e3062e5d0ee94b4da817378"} Jul 15 13:10:37.215333 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:37.215302 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-2fjtw_c783ddea-55cf-4263-8db7-d8d43cea99f2/kube-storage-version-migrator-operator/0.log" Jul 15 13:10:38.424127 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:38.424089 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" event={"ID":"88d80b94-5bd0-414c-ba80-343054aae400","Type":"ContainerStarted","Data":"efe28f5099421e961799c52ed20f0f09472e7990f59aea7cd1769029dc1f9750"} Jul 15 13:10:38.458330 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:38.458270 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-fh6zv" podStartSLOduration=1.419455745 podStartE2EDuration="3.458250651s" totalImagesPullingTime="2.038794906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:10:38.456373331 +0000 UTC m=+244.411827927" watchObservedRunningTime="2026-07-15 13:10:38.458250651 +0000 UTC m=+244.413705220" Jul 15 13:10:39.555144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:39.555110 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:39.555631 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:39.555220 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:39.555631 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:39.555317 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 15 13:10:39.555631 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:39.555390 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:47.555368692 +0000 UTC m=+253.510823253 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : configmap "kube-rbac-proxy" not found Jul 15 13:10:39.555631 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:39.555322 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:39.555631 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:39.555502 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:47.55546219 +0000 UTC m=+253.510916736 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:42.982986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:42.982946 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:42.983392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:42.983043 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:42.983392 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:42.983130 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:58.983111317 +0000 UTC m=+264.938565859 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : configmap references non-existent config key: service-ca.crt Jul 15 13:10:42.983392 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:42.983175 2564 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 15 13:10:42.983392 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:42.983236 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs podName:8dbe8e4b-d986-49e1-a374-01d806777597 nodeName:}" failed. No retries permitted until 2026-07-15 13:10:58.983223057 +0000 UTC m=+264.938677598 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs") pod "router-default-66dd8dbb64-8vkwx" (UID: "8dbe8e4b-d986-49e1-a374-01d806777597") : secret "router-metrics-certs-default" not found Jul 15 13:10:43.083950 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.083921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:43.086361 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.086329 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/f6ecc040-d4fc-4de9-90ba-660e77e32b54-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-2qsqp\" (UID: \"f6ecc040-d4fc-4de9-90ba-660e77e32b54\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:43.184424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.184388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:43.186771 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.186747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"image-registry-8698fd8df6-jzrgw\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:43.199622 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.199601 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-7dhmn\"" Jul 15 13:10:43.208518 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.208506 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" Jul 15 13:10:43.298029 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.297374 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j4np7\"" Jul 15 13:10:43.305283 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.305260 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:43.326457 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.326398 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp"] Jul 15 13:10:43.430515 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.430493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8698fd8df6-jzrgw"] Jul 15 13:10:43.432710 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:10:43.432683 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c0ef71e_cd6b_4c2f_96b3_92d22906c837.slice/crio-3758b2de4a94eff6015617fbfa0a97b77615b43cad379cfd585e6068fa5df6f3 WatchSource:0}: Error finding container 3758b2de4a94eff6015617fbfa0a97b77615b43cad379cfd585e6068fa5df6f3: Status 404 returned error can't find the container with id 3758b2de4a94eff6015617fbfa0a97b77615b43cad379cfd585e6068fa5df6f3 Jul 15 13:10:43.442189 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.442164 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" event={"ID":"f6ecc040-d4fc-4de9-90ba-660e77e32b54","Type":"ContainerStarted","Data":"6639a2e12844eb4bdebd312fde3476c384acd2920688aad81d831cbdb2be97e8"} Jul 15 13:10:43.443276 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:43.443256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" event={"ID":"4c0ef71e-cd6b-4c2f-96b3-92d22906c837","Type":"ContainerStarted","Data":"3758b2de4a94eff6015617fbfa0a97b77615b43cad379cfd585e6068fa5df6f3"} Jul 15 13:10:44.448458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:44.448212 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" event={"ID":"4c0ef71e-cd6b-4c2f-96b3-92d22906c837","Type":"ContainerStarted","Data":"dc706ac04e2c3aa0a7c13b3c5f0aca3003022a8a17acda59d6925b7040abdb63"} Jul 15 13:10:44.448458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:44.448329 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:10:45.453461 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:45.453416 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" event={"ID":"f6ecc040-d4fc-4de9-90ba-660e77e32b54","Type":"ContainerStarted","Data":"811960aa709e924a52d761e23bdfc8cb2951ae898344155c71329c9fa8fbfa39"} Jul 15 13:10:45.453926 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:45.453505 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" event={"ID":"f6ecc040-d4fc-4de9-90ba-660e77e32b54","Type":"ContainerStarted","Data":"732c2c4bb3c932c8eef95b6bb9a637bd50a99f98b62d973999509fbe20426795"} Jul 15 13:10:45.472089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:45.472046 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-2qsqp" podStartSLOduration=16.604458324 podStartE2EDuration="18.472030234s" totalImagesPullingTime="1.86757191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:10:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:10:45.471302037 +0000 UTC m=+251.426756607" watchObservedRunningTime="2026-07-15 13:10:45.472030234 +0000 UTC m=+251.427484841" Jul 15 13:10:45.472224 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:45.472125 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" podStartSLOduration=18.472119267 podStartE2EDuration="18.472119267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:10:44.46789587 +0000 UTC m=+250.423350461" watchObservedRunningTime="2026-07-15 13:10:45.472119267 +0000 UTC m=+251.427573829" Jul 15 13:10:46.414201 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:46.414162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:10:46.416447 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:46.416421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dea320b5-7fcb-48e0-bf9e-b9475cfb141a-metrics-certs\") pod \"network-metrics-daemon-5vjbj\" (UID: \"dea320b5-7fcb-48e0-bf9e-b9475cfb141a\") " pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:10:46.555430 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:46.555399 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8tgt2\"" Jul 15 13:10:46.562764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:46.562740 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5vjbj" Jul 15 13:10:46.694268 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:46.694187 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5vjbj"] Jul 15 13:10:46.700904 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:10:46.700883 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea320b5_7fcb_48e0_bf9e_b9475cfb141a.slice/crio-3737f7ab364024d6a0554e9e64557c31fc815a6b7bbb3b3f25d31db800c78f3b WatchSource:0}: Error finding container 3737f7ab364024d6a0554e9e64557c31fc815a6b7bbb3b3f25d31db800c78f3b: Status 404 returned error can't find the container with id 3737f7ab364024d6a0554e9e64557c31fc815a6b7bbb3b3f25d31db800c78f3b Jul 15 13:10:47.460515 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:47.460463 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5vjbj" event={"ID":"dea320b5-7fcb-48e0-bf9e-b9475cfb141a","Type":"ContainerStarted","Data":"3737f7ab364024d6a0554e9e64557c31fc815a6b7bbb3b3f25d31db800c78f3b"} Jul 15 13:10:47.625859 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:47.625777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:47.626239 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:47.625868 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:47.626239 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:47.625989 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 15 13:10:47.626239 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:10:47.626060 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls podName:499499f3-41c2-4699-98d0-69c986e2faf3 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:03.626040465 +0000 UTC m=+269.581495028 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8hqsl" (UID: "499499f3-41c2-4699-98d0-69c986e2faf3") : secret "insights-runtime-extractor-tls" not found Jul 15 13:10:47.626397 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:47.626342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/499499f3-41c2-4699-98d0-69c986e2faf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:10:48.465194 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:48.465154 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5vjbj" event={"ID":"dea320b5-7fcb-48e0-bf9e-b9475cfb141a","Type":"ContainerStarted","Data":"082be0501eed81fa7ec5bab4d197291d960ae2b844a40c8eed18bb30924b9ac3"} Jul 15 13:10:48.465350 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:48.465219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5vjbj" event={"ID":"dea320b5-7fcb-48e0-bf9e-b9475cfb141a","Type":"ContainerStarted","Data":"e0ed29c470eb271c5c7d94da923946ea25ece4bdd4a9f874d542880061b6ba54"} Jul 15 13:10:48.484804 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:48.484754 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5vjbj" podStartSLOduration=252.590761411 podStartE2EDuration="4m13.484738761s" totalImagesPullingTime="893.97735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:06:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:10:48.484240894 +0000 UTC m=+254.439695456" watchObservedRunningTime="2026-07-15 13:10:48.484738761 +0000 UTC m=+254.440193323" Jul 15 13:10:56.214986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:56.214911 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8698fd8df6-jzrgw"] Jul 15 13:10:59.006736 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.006694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:59.007248 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.006772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:59.007448 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.007423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8dbe8e4b-d986-49e1-a374-01d806777597-service-ca-bundle\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:59.009015 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.008994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8dbe8e4b-d986-49e1-a374-01d806777597-metrics-certs\") pod \"router-default-66dd8dbb64-8vkwx\" (UID: \"8dbe8e4b-d986-49e1-a374-01d806777597\") " pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:59.306545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.306514 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-25vfg\"" Jul 15 13:10:59.314523 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.314496 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:10:59.473945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.473920 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66dd8dbb64-8vkwx"] Jul 15 13:10:59.476217 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:10:59.476184 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dbe8e4b_d986_49e1_a374_01d806777597.slice/crio-0121653a24593128b2887ea4ecc53e24b7552ee96a61f720bf004ba40a289e44 WatchSource:0}: Error finding container 0121653a24593128b2887ea4ecc53e24b7552ee96a61f720bf004ba40a289e44: Status 404 returned error can't find the container with id 0121653a24593128b2887ea4ecc53e24b7552ee96a61f720bf004ba40a289e44 Jul 15 13:10:59.500212 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:10:59.500182 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" event={"ID":"8dbe8e4b-d986-49e1-a374-01d806777597","Type":"ContainerStarted","Data":"0121653a24593128b2887ea4ecc53e24b7552ee96a61f720bf004ba40a289e44"} Jul 15 13:11:00.504233 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:00.504196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" event={"ID":"8dbe8e4b-d986-49e1-a374-01d806777597","Type":"ContainerStarted","Data":"0ecd9c4c32ed1be7467d45c266a52892b37af6190d25d3611226a870b0db2442"} Jul 15 13:11:00.523871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:00.523825 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" podStartSLOduration=33.523808175 podStartE2EDuration="33.523808175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:00.522985678 +0000 UTC m=+266.478440255" watchObservedRunningTime="2026-07-15 13:11:00.523808175 +0000 UTC m=+266.479262738" Jul 15 13:11:01.315498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.315440 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:11:01.317947 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.317923 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:11:01.507740 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.507710 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:11:01.508873 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.508855 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66dd8dbb64-8vkwx" Jul 15 13:11:01.784587 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.784556 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n"] Jul 15 13:11:01.786548 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.786533 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:01.789036 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.789012 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 15 13:11:01.789152 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.789044 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-6vt96\"" Jul 15 13:11:01.795990 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.795968 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n"] Jul 15 13:11:01.929837 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:01.929748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a032e11a-e613-45d0-9cfd-db25e81633e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-glr7n\" (UID: \"a032e11a-e613-45d0-9cfd-db25e81633e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:02.030777 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:02.030741 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a032e11a-e613-45d0-9cfd-db25e81633e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-glr7n\" (UID: \"a032e11a-e613-45d0-9cfd-db25e81633e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:02.030921 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:02.030884 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 15 13:11:02.030961 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:02.030945 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a032e11a-e613-45d0-9cfd-db25e81633e7-tls-certificates podName:a032e11a-e613-45d0-9cfd-db25e81633e7 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:02.530929129 +0000 UTC m=+268.486383671 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a032e11a-e613-45d0-9cfd-db25e81633e7-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-glr7n" (UID: "a032e11a-e613-45d0-9cfd-db25e81633e7") : secret "prometheus-operator-admission-webhook-tls" not found Jul 15 13:11:02.533463 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:02.533428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a032e11a-e613-45d0-9cfd-db25e81633e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-glr7n\" (UID: \"a032e11a-e613-45d0-9cfd-db25e81633e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:02.538108 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:02.538086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a032e11a-e613-45d0-9cfd-db25e81633e7-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-glr7n\" (UID: \"a032e11a-e613-45d0-9cfd-db25e81633e7\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:02.694811 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:02.694766 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:02.816724 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:02.816625 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n"] Jul 15 13:11:02.819264 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:02.819242 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda032e11a_e613_45d0_9cfd_db25e81633e7.slice/crio-c1a56375a5fc76dc1bf0d98fa567cc11d65bb04d944b762a0a746d73d9d543e2 WatchSource:0}: Error finding container c1a56375a5fc76dc1bf0d98fa567cc11d65bb04d944b762a0a746d73d9d543e2: Status 404 returned error can't find the container with id c1a56375a5fc76dc1bf0d98fa567cc11d65bb04d944b762a0a746d73d9d543e2 Jul 15 13:11:03.515851 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.515816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" event={"ID":"a032e11a-e613-45d0-9cfd-db25e81633e7","Type":"ContainerStarted","Data":"c1a56375a5fc76dc1bf0d98fa567cc11d65bb04d944b762a0a746d73d9d543e2"} Jul 15 13:11:03.639619 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.639585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:11:03.641963 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.641937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/499499f3-41c2-4699-98d0-69c986e2faf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8hqsl\" (UID: \"499499f3-41c2-4699-98d0-69c986e2faf3\") " pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:11:03.855383 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.855351 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7bcf7bcdbf-lx4kt"] Jul 15 13:11:03.857452 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.857430 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.859862 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.859842 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 13:11:03.860972 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.860933 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f5tck\"" Jul 15 13:11:03.860972 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.860942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 13:11:03.860972 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.860957 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 13:11:03.861183 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.860941 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 13:11:03.861183 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.860947 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 13:11:03.861183 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.860933 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 13:11:03.861864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.861846 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 13:11:03.865210 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.865191 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 13:11:03.868144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.868123 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcf7bcdbf-lx4kt"] Jul 15 13:11:03.943034 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943007 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-serving-cert\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943139 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-oauth-serving-cert\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943139 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq94j\" (UniqueName: \"kubernetes.io/projected/a5138255-6be6-4595-a07e-636121c55035-kube-api-access-mq94j\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-trusted-ca-bundle\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943163 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-service-ca\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-oauth-config\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943360 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943245 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-console-config\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:03.943783 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.943768 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m9psk\"" Jul 15 13:11:03.951657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:03.951639 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8hqsl" Jul 15 13:11:04.043696 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043667 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-console-config\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.043867 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043718 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-serving-cert\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.043867 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043747 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-oauth-serving-cert\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.043867 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq94j\" (UniqueName: \"kubernetes.io/projected/a5138255-6be6-4595-a07e-636121c55035-kube-api-access-mq94j\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.043867 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043784 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-trusted-ca-bundle\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.043867 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043836 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-service-ca\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.044158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.043873 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-oauth-config\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.044440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.044416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-oauth-serving-cert\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.044568 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.044455 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-console-config\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.044568 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.044555 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-service-ca\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.044818 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.044795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-trusted-ca-bundle\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.046346 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.046324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-oauth-config\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.046807 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.046786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-serving-cert\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.053601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.053578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq94j\" (UniqueName: \"kubernetes.io/projected/a5138255-6be6-4595-a07e-636121c55035-kube-api-access-mq94j\") pod \"console-7bcf7bcdbf-lx4kt\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.100682 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.100653 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8hqsl"] Jul 15 13:11:04.103065 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:04.103034 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod499499f3_41c2_4699_98d0_69c986e2faf3.slice/crio-033cef89821d95e19dc2aa7944deac6f543829dc36ef79c1e4a29ab59c343766 WatchSource:0}: Error finding container 033cef89821d95e19dc2aa7944deac6f543829dc36ef79c1e4a29ab59c343766: Status 404 returned error can't find the container with id 033cef89821d95e19dc2aa7944deac6f543829dc36ef79c1e4a29ab59c343766 Jul 15 13:11:04.168052 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.168014 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:04.296075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.296050 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7bcf7bcdbf-lx4kt"] Jul 15 13:11:04.298398 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:04.298374 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5138255_6be6_4595_a07e_636121c55035.slice/crio-5a73159e7a9bdcf3bf797a86c3d7f14d4b72cbd83b7409f1939d47ebfaa3790a WatchSource:0}: Error finding container 5a73159e7a9bdcf3bf797a86c3d7f14d4b72cbd83b7409f1939d47ebfaa3790a: Status 404 returned error can't find the container with id 5a73159e7a9bdcf3bf797a86c3d7f14d4b72cbd83b7409f1939d47ebfaa3790a Jul 15 13:11:04.520097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.520016 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hqsl" event={"ID":"499499f3-41c2-4699-98d0-69c986e2faf3","Type":"ContainerStarted","Data":"033cef89821d95e19dc2aa7944deac6f543829dc36ef79c1e4a29ab59c343766"} Jul 15 13:11:04.521234 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.521209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" event={"ID":"a032e11a-e613-45d0-9cfd-db25e81633e7","Type":"ContainerStarted","Data":"aca72aa62daf540adf593c92f0272ceac6ed0feffaf2fb3401ed254264259707"} Jul 15 13:11:04.521446 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.521404 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:04.522389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.522357 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcf7bcdbf-lx4kt" event={"ID":"a5138255-6be6-4595-a07e-636121c55035","Type":"ContainerStarted","Data":"5a73159e7a9bdcf3bf797a86c3d7f14d4b72cbd83b7409f1939d47ebfaa3790a"} Jul 15 13:11:04.525750 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.525729 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" Jul 15 13:11:04.543638 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:04.543599 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-glr7n" podStartSLOduration=2.489214304 podStartE2EDuration="3.543583277s" totalImagesPullingTime="1.054368973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:04.542847704 +0000 UTC m=+270.498302254" watchObservedRunningTime="2026-07-15 13:11:04.543583277 +0000 UTC m=+270.499037841" Jul 15 13:11:06.222175 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:06.222129 2564 patch_prober.go:28] interesting pod/image-registry-8698fd8df6-jzrgw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 13:11:06.222619 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:06.222186 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:11:09.541338 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:09.541308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hqsl" event={"ID":"499499f3-41c2-4699-98d0-69c986e2faf3","Type":"ContainerStarted","Data":"103138bbcd57bd7d80a1c0164b56b209e9bec4ab1f29ade9f49720569ea39e33"} Jul 15 13:11:11.550956 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:11.550917 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcf7bcdbf-lx4kt" event={"ID":"a5138255-6be6-4595-a07e-636121c55035","Type":"ContainerStarted","Data":"07223b2de88370ee503c65f7bd65d8f5c3adce2970776195cf90114e73d7263f"} Jul 15 13:11:11.569147 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:11.569092 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7bcf7bcdbf-lx4kt" podStartSLOduration=2.028290223 podStartE2EDuration="8.569074425s" totalImagesPullingTime="6.540784202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:11.568631086 +0000 UTC m=+277.524085661" watchObservedRunningTime="2026-07-15 13:11:11.569074425 +0000 UTC m=+277.524528990" Jul 15 13:11:13.559664 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:13.559622 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hqsl" event={"ID":"499499f3-41c2-4699-98d0-69c986e2faf3","Type":"ContainerStarted","Data":"196c2280a542c810df22be7cb190e44235faab8d4fcddfc5d38cb78e677c0772"} Jul 15 13:11:14.168654 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:14.168615 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:14.168973 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:14.168946 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:14.170266 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:14.170241 2564 patch_prober.go:28] interesting pod/console-7bcf7bcdbf-lx4kt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body= Jul 15 13:11:14.170372 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:14.170290 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7bcf7bcdbf-lx4kt" podUID="a5138255-6be6-4595-a07e-636121c55035" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" Jul 15 13:11:16.219672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:16.219638 2564 patch_prober.go:28] interesting pod/image-registry-8698fd8df6-jzrgw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 15 13:11:16.220051 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:16.219694 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:11:16.571718 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:16.571681 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8hqsl" event={"ID":"499499f3-41c2-4699-98d0-69c986e2faf3","Type":"ContainerStarted","Data":"7890df76d2b2263b68d27bda6d49464ba1f756faccff420dc612f41449475e6f"} Jul 15 13:11:16.592587 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:16.592539 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8hqsl" podStartSLOduration=33.575459567 podStartE2EDuration="45.592524314s" totalImagesPullingTime="12.017064747s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:10:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:16.591372809 +0000 UTC m=+282.546827397" watchObservedRunningTime="2026-07-15 13:11:16.592524314 +0000 UTC m=+282.547978876" Jul 15 13:11:17.207720 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:17.207677 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-wvk9l" podUID="65a06378-bbbd-412e-95fe-21d586ec36ea" Jul 15 13:11:17.575328 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:17.575297 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvk9l" Jul 15 13:11:20.164719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:20.164680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:11:20.169085 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:20.168002 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/65a06378-bbbd-412e-95fe-21d586ec36ea-metrics-tls\") pod \"dns-default-wvk9l\" (UID: \"65a06378-bbbd-412e-95fe-21d586ec36ea\") " pod="openshift-dns/dns-default-wvk9l" Jul 15 13:11:20.278646 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:20.278617 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-txt24\"" Jul 15 13:11:20.286955 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:20.286940 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wvk9l" Jul 15 13:11:20.411668 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:20.411638 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wvk9l"] Jul 15 13:11:20.414304 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:20.414276 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a06378_bbbd_412e_95fe_21d586ec36ea.slice/crio-8616adabdac038654ac708cdbac16acb6dd5a767d27853d0a9e014e802a3df47 WatchSource:0}: Error finding container 8616adabdac038654ac708cdbac16acb6dd5a767d27853d0a9e014e802a3df47: Status 404 returned error can't find the container with id 8616adabdac038654ac708cdbac16acb6dd5a767d27853d0a9e014e802a3df47 Jul 15 13:11:20.586145 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:20.586116 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvk9l" event={"ID":"65a06378-bbbd-412e-95fe-21d586ec36ea","Type":"ContainerStarted","Data":"8616adabdac038654ac708cdbac16acb6dd5a767d27853d0a9e014e802a3df47"} Jul 15 13:11:21.234519 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.234457 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerName="registry" containerID="cri-o://dc706ac04e2c3aa0a7c13b3c5f0aca3003022a8a17acda59d6925b7040abdb63" gracePeriod=30 Jul 15 13:11:21.591423 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.591391 2564 generic.go:356] "Generic (PLEG): container finished" podID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerID="dc706ac04e2c3aa0a7c13b3c5f0aca3003022a8a17acda59d6925b7040abdb63" exitCode=0 Jul 15 13:11:21.591601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.591444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" event={"ID":"4c0ef71e-cd6b-4c2f-96b3-92d22906c837","Type":"ContainerDied","Data":"dc706ac04e2c3aa0a7c13b3c5f0aca3003022a8a17acda59d6925b7040abdb63"} Jul 15 13:11:21.835536 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.835513 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:11:21.880574 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-bound-sa-token\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880663 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880594 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78qrt\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-kube-api-access-78qrt\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880663 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880628 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880777 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880759 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-installation-pull-secrets\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880843 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880827 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-image-registry-private-configuration\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880879 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-ca-trust-extracted\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880888 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-trusted-ca\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.880980 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.880938 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-certificates\") pod \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\" (UID: \"4c0ef71e-cd6b-4c2f-96b3-92d22906c837\") " Jul 15 13:11:21.881498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.881416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:11:21.881498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.881443 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:11:21.883207 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.883181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:21.883306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.883228 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-kube-api-access-78qrt" (OuterVolumeSpecName: "kube-api-access-78qrt") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "kube-api-access-78qrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:21.883306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.883271 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:11:21.883306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.883292 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:11:21.883527 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.883506 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:11:21.890333 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.890310 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4c0ef71e-cd6b-4c2f-96b3-92d22906c837" (UID: "4c0ef71e-cd6b-4c2f-96b3-92d22906c837"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:11:21.981994 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.981972 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-image-registry-private-configuration\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982103 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982061 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-ca-trust-extracted\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982103 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982081 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-trusted-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982103 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982094 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-certificates\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982236 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982122 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-bound-sa-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982236 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982137 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78qrt\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-kube-api-access-78qrt\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982236 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982149 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-registry-tls\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:21.982236 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:21.982163 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4c0ef71e-cd6b-4c2f-96b3-92d22906c837-installation-pull-secrets\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:11:22.596744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.596713 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" Jul 15 13:11:22.596744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.596730 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8698fd8df6-jzrgw" event={"ID":"4c0ef71e-cd6b-4c2f-96b3-92d22906c837","Type":"ContainerDied","Data":"3758b2de4a94eff6015617fbfa0a97b77615b43cad379cfd585e6068fa5df6f3"} Jul 15 13:11:22.597324 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.596788 2564 scope.go:117] "RemoveContainer" containerID="dc706ac04e2c3aa0a7c13b3c5f0aca3003022a8a17acda59d6925b7040abdb63" Jul 15 13:11:22.598636 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.598612 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvk9l" event={"ID":"65a06378-bbbd-412e-95fe-21d586ec36ea","Type":"ContainerStarted","Data":"2a1ee84f106d3a58d9263f77c95787e3283e2c0bfc93d54e8c1573742c41f119"} Jul 15 13:11:22.598740 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.598658 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wvk9l" event={"ID":"65a06378-bbbd-412e-95fe-21d586ec36ea","Type":"ContainerStarted","Data":"99768429f9fbe57394bf1aad7441d1dc103d1a60fc0b88f7b47b1906862ff755"} Jul 15 13:11:22.598740 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.598681 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wvk9l" Jul 15 13:11:22.621770 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.621728 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wvk9l" podStartSLOduration=251.160528529 podStartE2EDuration="4m12.621715631s" totalImagesPullingTime="1.461187102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:07:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:22.620730967 +0000 UTC m=+288.576185524" watchObservedRunningTime="2026-07-15 13:11:22.621715631 +0000 UTC m=+288.577170241" Jul 15 13:11:22.636274 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.636254 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8698fd8df6-jzrgw"] Jul 15 13:11:22.639655 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.639634 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8698fd8df6-jzrgw"] Jul 15 13:11:22.656170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:22.656147 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" path="/var/lib/kubelet/pods/4c0ef71e-cd6b-4c2f-96b3-92d22906c837/volumes" Jul 15 13:11:24.168575 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:24.168538 2564 patch_prober.go:28] interesting pod/console-7bcf7bcdbf-lx4kt container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" start-of-body= Jul 15 13:11:24.168956 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:24.168604 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-7bcf7bcdbf-lx4kt" podUID="a5138255-6be6-4595-a07e-636121c55035" containerName="console" probeResult="failure" output="Get \"https://10.134.0.16:8443/health\": dial tcp 10.134.0.16:8443: connect: connection refused" Jul 15 13:11:31.855318 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.855278 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55"] Jul 15 13:11:31.855707 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.855642 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerName="registry" Jul 15 13:11:31.855707 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.855658 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerName="registry" Jul 15 13:11:31.855780 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.855723 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4c0ef71e-cd6b-4c2f-96b3-92d22906c837" containerName="registry" Jul 15 13:11:31.858089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.858067 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:31.862986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.862963 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 15 13:11:31.863672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.863653 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 15 13:11:31.863780 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.863663 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 15 13:11:31.863849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.863812 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 15 13:11:31.863849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.863839 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d5pp5\"" Jul 15 13:11:31.863964 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.863667 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 15 13:11:31.868067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.868047 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-glk2f"] Jul 15 13:11:31.869944 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.869930 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:31.872658 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.872638 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 15 13:11:31.872981 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.872966 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 15 13:11:31.873064 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.873003 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 15 13:11:31.873221 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.873204 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-qrzpx\"" Jul 15 13:11:31.875262 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.875244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55"] Jul 15 13:11:31.891160 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.891129 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-glk2f"] Jul 15 13:11:31.904465 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.904448 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-c2cwm"] Jul 15 13:11:31.906541 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.906527 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.910106 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.910089 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 15 13:11:31.910507 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.910454 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fx726\"" Jul 15 13:11:31.910507 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.910461 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 15 13:11:31.910786 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.910760 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 15 13:11:31.944649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944625 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:31.944760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-metrics-client-ca\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.944760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944685 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.944760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-textfile\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944782 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-wtmp\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77ffc\" (UniqueName: \"kubernetes.io/projected/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-api-access-77ffc\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-root\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944875 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-sys\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.944918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944936 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf69ac6-c3bb-4579-94f7-93d18204abb2-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fww5\" (UniqueName: \"kubernetes.io/projected/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-kube-api-access-7fww5\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.944979 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-tls\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.945000 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.945044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.945085 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6rpd\" (UniqueName: \"kubernetes.io/projected/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-kube-api-access-t6rpd\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.945118 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:31.945225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:31.945134 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fcf69ac6-c3bb-4579-94f7-93d18204abb2-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.046335 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.046304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fww5\" (UniqueName: \"kubernetes.io/projected/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-kube-api-access-7fww5\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.046509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.046352 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-tls\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.046509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.046410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.046509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.046456 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.046672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.046560 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6rpd\" (UniqueName: \"kubernetes.io/projected/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-kube-api-access-t6rpd\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047102 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:32.046948 2564 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 15 13:11:32.047219 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:32.047153 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-tls podName:f2aeca5e-acaf-415c-aa19-44ebf58cbd4d nodeName:}" failed. No retries permitted until 2026-07-15 13:11:32.547127268 +0000 UTC m=+298.502581822 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-tls") pod "node-exporter-c2cwm" (UID: "f2aeca5e-acaf-415c-aa19-44ebf58cbd4d") : secret "node-exporter-tls" not found Jul 15 13:11:32.047219 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.047347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047231 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fcf69ac6-c3bb-4579-94f7-93d18204abb2-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.047347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047249 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.047347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047263 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.047347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047291 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-metrics-client-ca\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-metrics-client-ca\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.047347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047333 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047363 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-textfile\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-wtmp\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047415 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77ffc\" (UniqueName: \"kubernetes.io/projected/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-api-access-77ffc\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047461 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-root\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047517 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-sys\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047573 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.047656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047604 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf69ac6-c3bb-4579-94f7-93d18204abb2-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.048083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.047935 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/fcf69ac6-c3bb-4579-94f7-93d18204abb2-volume-directive-shadow\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.048083 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:32.047960 2564 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 15 13:11:32.048083 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:11:32.048049 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-tls podName:fcf69ac6-c3bb-4579-94f7-93d18204abb2 nodeName:}" failed. No retries permitted until 2026-07-15 13:11:32.548019685 +0000 UTC m=+298.503474246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-tls") pod "kube-state-metrics-785778ff74-glk2f" (UID: "fcf69ac6-c3bb-4579-94f7-93d18204abb2") : secret "kube-state-metrics-tls" not found Jul 15 13:11:32.048234 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.048210 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcf69ac6-c3bb-4579-94f7-93d18204abb2-metrics-client-ca\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.048290 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.048271 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-sys\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.048849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.048524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-wtmp\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.048849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.048626 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-metrics-client-ca\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.048849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.048806 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-root\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.049083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.049061 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-accelerators-collector-config\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.049388 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.049368 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-textfile\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.050215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.050190 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.050853 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.050818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-openshift-state-metrics-tls\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.051907 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.051878 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.052136 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.052116 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.059218 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.059193 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fww5\" (UniqueName: \"kubernetes.io/projected/25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba-kube-api-access-7fww5\") pod \"openshift-state-metrics-84bd68b896-6qg55\" (UID: \"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba\") " pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.062706 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.062684 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6rpd\" (UniqueName: \"kubernetes.io/projected/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-kube-api-access-t6rpd\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.066402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.066374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77ffc\" (UniqueName: \"kubernetes.io/projected/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-api-access-77ffc\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.169605 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.169528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" Jul 15 13:11:32.307030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.307000 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55"] Jul 15 13:11:32.310385 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:32.310355 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ca13b1_d6d0_44e6_b8a7_89e0a1e46aba.slice/crio-df6d3bc5ff5742ef969302d011e9cd61eea4ea73e9a1155c15aef47bccca4316 WatchSource:0}: Error finding container df6d3bc5ff5742ef969302d011e9cd61eea4ea73e9a1155c15aef47bccca4316: Status 404 returned error can't find the container with id df6d3bc5ff5742ef969302d011e9cd61eea4ea73e9a1155c15aef47bccca4316 Jul 15 13:11:32.552438 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.552413 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-tls\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.552567 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.552464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.554549 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.554524 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f2aeca5e-acaf-415c-aa19-44ebf58cbd4d-node-exporter-tls\") pod \"node-exporter-c2cwm\" (UID: \"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d\") " pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.554682 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.554666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcf69ac6-c3bb-4579-94f7-93d18204abb2-kube-state-metrics-tls\") pod \"kube-state-metrics-785778ff74-glk2f\" (UID: \"fcf69ac6-c3bb-4579-94f7-93d18204abb2\") " pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.605808 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.605789 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wvk9l" Jul 15 13:11:32.637252 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.637219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" event={"ID":"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba","Type":"ContainerStarted","Data":"a3768fa668acd582262fbba84e0c656444390f421f0c39a703a49a748c89a016"} Jul 15 13:11:32.637395 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.637287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" event={"ID":"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba","Type":"ContainerStarted","Data":"758186c01d47cc5480db14fd969e262b445b724f8f307fd2ab6af6394ef86c43"} Jul 15 13:11:32.637395 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.637316 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" event={"ID":"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba","Type":"ContainerStarted","Data":"df6d3bc5ff5742ef969302d011e9cd61eea4ea73e9a1155c15aef47bccca4316"} Jul 15 13:11:32.779325 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.779255 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" Jul 15 13:11:32.814902 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.814876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-c2cwm" Jul 15 13:11:32.825253 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:32.825222 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2aeca5e_acaf_415c_aa19_44ebf58cbd4d.slice/crio-5d0eabbae9cb3804a2650dc565bcd3fb9a4dbefb5de78cd62aa67e31959f1014 WatchSource:0}: Error finding container 5d0eabbae9cb3804a2650dc565bcd3fb9a4dbefb5de78cd62aa67e31959f1014: Status 404 returned error can't find the container with id 5d0eabbae9cb3804a2650dc565bcd3fb9a4dbefb5de78cd62aa67e31959f1014 Jul 15 13:11:32.919184 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:32.919102 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-785778ff74-glk2f"] Jul 15 13:11:32.921726 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:32.921697 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf69ac6_c3bb_4579_94f7_93d18204abb2.slice/crio-e1995ecb98781ae60f20aed8157cfdf7f7ffd5e18d40898da3f1692e8ed5ec68 WatchSource:0}: Error finding container e1995ecb98781ae60f20aed8157cfdf7f7ffd5e18d40898da3f1692e8ed5ec68: Status 404 returned error can't find the container with id e1995ecb98781ae60f20aed8157cfdf7f7ffd5e18d40898da3f1692e8ed5ec68 Jul 15 13:11:33.039622 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.039592 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 13:11:33.042876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.042854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.045616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.045593 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-kx6pn\"" Jul 15 13:11:33.045886 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.045868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 15 13:11:33.045971 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.045868 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 15 13:11:33.045971 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.045928 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 15 13:11:33.046459 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.046244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 15 13:11:33.046459 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.046395 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 15 13:11:33.046644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.045879 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 15 13:11:33.047059 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.046827 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 15 13:11:33.048990 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.048970 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 15 13:11:33.053349 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.053328 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 15 13:11:33.061974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.061946 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 13:11:33.158215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158233 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5252322-2e18-42fb-b72b-392ef40ceacc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158255 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5252322-2e18-42fb-b72b-392ef40ceacc-config-out\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5252322-2e18-42fb-b72b-392ef40ceacc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158399 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5252322-2e18-42fb-b72b-392ef40ceacc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-web-config\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158455 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rb2r\" (UniqueName: \"kubernetes.io/projected/d5252322-2e18-42fb-b72b-392ef40ceacc-kube-api-access-5rb2r\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158517 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5252322-2e18-42fb-b72b-392ef40ceacc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158542 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158573 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158593 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.158620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.158613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259048 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259192 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259058 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259192 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259084 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5252322-2e18-42fb-b72b-392ef40ceacc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259192 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5252322-2e18-42fb-b72b-392ef40ceacc-config-out\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259192 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5252322-2e18-42fb-b72b-392ef40ceacc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259390 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259197 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5252322-2e18-42fb-b72b-392ef40ceacc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259390 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-web-config\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259779 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259572 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rb2r\" (UniqueName: \"kubernetes.io/projected/d5252322-2e18-42fb-b72b-392ef40ceacc-kube-api-access-5rb2r\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.259779 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5252322-2e18-42fb-b72b-392ef40ceacc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.260277 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.259695 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.260383 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.260307 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.260446 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.260405 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.260728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.260708 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.262140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d5252322-2e18-42fb-b72b-392ef40ceacc-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.262566 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5252322-2e18-42fb-b72b-392ef40ceacc-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.262588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d5252322-2e18-42fb-b72b-392ef40ceacc-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.262736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.262926 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d5252322-2e18-42fb-b72b-392ef40ceacc-config-out\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.263033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-config-volume\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.263801 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.263674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d5252322-2e18-42fb-b72b-392ef40ceacc-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.267540 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.265203 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-web-config\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.267540 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.265654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.267540 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.266622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.269607 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.269583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.270557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.270533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d5252322-2e18-42fb-b72b-392ef40ceacc-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.272719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.272696 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rb2r\" (UniqueName: \"kubernetes.io/projected/d5252322-2e18-42fb-b72b-392ef40ceacc-kube-api-access-5rb2r\") pod \"alertmanager-main-0\" (UID: \"d5252322-2e18-42fb-b72b-392ef40ceacc\") " pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.376411 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.376337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 15 13:11:33.581909 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.581882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 15 13:11:33.585245 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:33.585217 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5252322_2e18_42fb_b72b_392ef40ceacc.slice/crio-1722d7c8459fda0a51441f3cc68b640d9dc5ed7774bc6d2c91f0ecdfa89f6f01 WatchSource:0}: Error finding container 1722d7c8459fda0a51441f3cc68b640d9dc5ed7774bc6d2c91f0ecdfa89f6f01: Status 404 returned error can't find the container with id 1722d7c8459fda0a51441f3cc68b640d9dc5ed7774bc6d2c91f0ecdfa89f6f01 Jul 15 13:11:33.642192 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.642100 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" event={"ID":"fcf69ac6-c3bb-4579-94f7-93d18204abb2","Type":"ContainerStarted","Data":"e1995ecb98781ae60f20aed8157cfdf7f7ffd5e18d40898da3f1692e8ed5ec68"} Jul 15 13:11:33.644637 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.644571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" event={"ID":"25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba","Type":"ContainerStarted","Data":"5cad3009a7668c776d28e78780ffcd9121599693abb4e19adda8829f9d73c1b9"} Jul 15 13:11:33.645894 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.645860 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"1722d7c8459fda0a51441f3cc68b640d9dc5ed7774bc6d2c91f0ecdfa89f6f01"} Jul 15 13:11:33.647122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.647097 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2cwm" event={"ID":"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d","Type":"ContainerStarted","Data":"5d0eabbae9cb3804a2650dc565bcd3fb9a4dbefb5de78cd62aa67e31959f1014"} Jul 15 13:11:33.663529 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.663463 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-84bd68b896-6qg55" podStartSLOduration=1.620480052 podStartE2EDuration="2.663445412s" totalImagesPullingTime="1.04296536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:33.662420948 +0000 UTC m=+299.617875511" watchObservedRunningTime="2026-07-15 13:11:33.663445412 +0000 UTC m=+299.618899977" Jul 15 13:11:33.940813 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.940732 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6897b47b98-8v9z9"] Jul 15 13:11:33.944727 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.944706 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:33.947771 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.947746 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 15 13:11:33.947871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.947793 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-cu9nh9rvsnpf\"" Jul 15 13:11:33.947871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.947801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 15 13:11:33.947987 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.947753 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 15 13:11:33.947987 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.947745 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 15 13:11:33.948083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.948058 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-vsnn2\"" Jul 15 13:11:33.948083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.948067 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 15 13:11:33.960558 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:33.960533 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6897b47b98-8v9z9"] Jul 15 13:11:34.068991 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.068962 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069127 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.068999 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-grpc-tls\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069127 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.069021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069127 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.069103 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069298 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.069201 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069298 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.069250 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-tls\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069298 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.069281 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-metrics-client-ca\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.069425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.069330 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxz9h\" (UniqueName: \"kubernetes.io/projected/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-kube-api-access-qxz9h\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171255 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171359 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171289 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171359 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171329 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-tls\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171453 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171358 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-metrics-client-ca\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171453 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxz9h\" (UniqueName: \"kubernetes.io/projected/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-kube-api-access-qxz9h\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-grpc-tls\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.171583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.171522 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.172933 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.172907 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-metrics-client-ca\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.175451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.174449 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:34.175451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.174954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.177697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.176649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.177697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.176860 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-tls\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.177697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.177641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.177898 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.177875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.178432 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.178407 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-secret-grpc-tls\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.181920 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.181688 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:11:34.183919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.183884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxz9h\" (UniqueName: \"kubernetes.io/projected/f6aec2de-f43e-4b1f-aed1-fc326495ce6e-kube-api-access-qxz9h\") pod \"thanos-querier-6897b47b98-8v9z9\" (UID: \"f6aec2de-f43e-4b1f-aed1-fc326495ce6e\") " pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.256913 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.256221 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:34.433240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.433208 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6897b47b98-8v9z9"] Jul 15 13:11:34.441094 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:34.441063 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6aec2de_f43e_4b1f_aed1_fc326495ce6e.slice/crio-cebf1c850426f12654178c75a3af09f3d397534bf193ae3a9157820ef034ba68 WatchSource:0}: Error finding container cebf1c850426f12654178c75a3af09f3d397534bf193ae3a9157820ef034ba68: Status 404 returned error can't find the container with id cebf1c850426f12654178c75a3af09f3d397534bf193ae3a9157820ef034ba68 Jul 15 13:11:34.486158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.486099 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:11:34.486735 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.486714 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:11:34.493185 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.493172 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 15 13:11:34.657864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.657835 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"cebf1c850426f12654178c75a3af09f3d397534bf193ae3a9157820ef034ba68"} Jul 15 13:11:34.658022 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.657901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" event={"ID":"fcf69ac6-c3bb-4579-94f7-93d18204abb2","Type":"ContainerStarted","Data":"7fc0c548338cf105b583a770abeb941700928ea2ed80d0f7fdea5174c1d5cb77"} Jul 15 13:11:34.658022 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.657937 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" event={"ID":"fcf69ac6-c3bb-4579-94f7-93d18204abb2","Type":"ContainerStarted","Data":"3f6b305973a59efb2be1bafb03c7ee58d323f540fae1034d8940a2f7660ea008"} Jul 15 13:11:34.658022 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.657964 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" event={"ID":"fcf69ac6-c3bb-4579-94f7-93d18204abb2","Type":"ContainerStarted","Data":"118c08a6fe51c8c6e64d6a69936f1d8851a867d2b72ecdedd0a11665626ffc2e"} Jul 15 13:11:34.677445 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:34.677290 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-785778ff74-glk2f" podStartSLOduration=2.50477622 podStartE2EDuration="3.677274469s" totalImagesPullingTime="1.172498249s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:34.676084077 +0000 UTC m=+300.631538641" watchObservedRunningTime="2026-07-15 13:11:34.677274469 +0000 UTC m=+300.632729082" Jul 15 13:11:35.661525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:35.661488 2564 generic.go:356] "Generic (PLEG): container finished" podID="d5252322-2e18-42fb-b72b-392ef40ceacc" containerID="cb22ac9de4ca07e660773e45df76111701f2bbc006c3002b3d921d56cdfefdb1" exitCode=0 Jul 15 13:11:35.661946 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:35.661563 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerDied","Data":"cb22ac9de4ca07e660773e45df76111701f2bbc006c3002b3d921d56cdfefdb1"} Jul 15 13:11:35.663223 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:35.663197 2564 generic.go:356] "Generic (PLEG): container finished" podID="c783ddea-55cf-4263-8db7-d8d43cea99f2" containerID="f0d103adb7f4c32a49857604cdba1b866aa6b4c0a71973720a13e6c14ca13c87" exitCode=0 Jul 15 13:11:35.663343 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:35.663298 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" event={"ID":"c783ddea-55cf-4263-8db7-d8d43cea99f2","Type":"ContainerDied","Data":"f0d103adb7f4c32a49857604cdba1b866aa6b4c0a71973720a13e6c14ca13c87"} Jul 15 13:11:35.663825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:35.663804 2564 scope.go:117] "RemoveContainer" containerID="f0d103adb7f4c32a49857604cdba1b866aa6b4c0a71973720a13e6c14ca13c87" Jul 15 13:11:36.155731 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.155694 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-78999c7f87-qbnww"] Jul 15 13:11:36.159130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.159108 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.161538 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.161515 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 15 13:11:36.161663 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.161565 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 15 13:11:36.161926 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.161903 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-95rp5glnbenv7\"" Jul 15 13:11:36.162772 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.162751 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gscjv\"" Jul 15 13:11:36.162885 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.162795 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 15 13:11:36.162885 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.162811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 15 13:11:36.169686 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.168977 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78999c7f87-qbnww"] Jul 15 13:11:36.293512 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293429 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-secret-metrics-server-tls\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.293512 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64g96\" (UniqueName: \"kubernetes.io/projected/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-kube-api-access-64g96\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.293759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-audit-log\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.293759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-metrics-server-audit-profiles\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.293759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293647 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-client-ca-bundle\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.293759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.293759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.293706 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-secret-metrics-server-client-certs\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.394796 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394758 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-audit-log\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.394976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-metrics-server-audit-profiles\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.394976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394846 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-client-ca-bundle\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.394976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394883 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.394976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-secret-metrics-server-client-certs\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.394976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-secret-metrics-server-tls\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.395235 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.394996 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64g96\" (UniqueName: \"kubernetes.io/projected/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-kube-api-access-64g96\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.395235 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.395200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-audit-log\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.396301 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.396272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-metrics-server-audit-profiles\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.396594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.396542 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.398259 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.398211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-client-ca-bundle\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.398364 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.398324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-secret-metrics-server-client-certs\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.398698 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.398672 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-secret-metrics-server-tls\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.403583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.403548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64g96\" (UniqueName: \"kubernetes.io/projected/1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970-kube-api-access-64g96\") pod \"metrics-server-78999c7f87-qbnww\" (UID: \"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970\") " pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.472640 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.472568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:36.671601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.671561 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-2fjtw" event={"ID":"c783ddea-55cf-4263-8db7-d8d43cea99f2","Type":"ContainerStarted","Data":"67ce066da0ac72e5fb317e1f51c38a3b52d45e832d3d8f1db29a94a59687a0c9"} Jul 15 13:11:36.811674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:36.811645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-78999c7f87-qbnww"] Jul 15 13:11:37.068930 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.068884 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c687b8b78-svdjj"] Jul 15 13:11:37.073018 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.072991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.075411 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.075385 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 15 13:11:37.075411 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.075394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 15 13:11:37.075595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.075412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 15 13:11:37.075836 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.075713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 15 13:11:37.075836 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.075760 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-frxhh\"" Jul 15 13:11:37.075997 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.075907 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 15 13:11:37.082184 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.081743 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 15 13:11:37.084930 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.084911 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c687b8b78-svdjj"] Jul 15 13:11:37.160555 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:37.160524 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9ff85e_f5c1_4fb5_919d_d3e7c18d9970.slice/crio-3f18566a7fdbac3d43e30ef16a46dea0d5adcd426530cda291ce69cb60025d10 WatchSource:0}: Error finding container 3f18566a7fdbac3d43e30ef16a46dea0d5adcd426530cda291ce69cb60025d10: Status 404 returned error can't find the container with id 3f18566a7fdbac3d43e30ef16a46dea0d5adcd426530cda291ce69cb60025d10 Jul 15 13:11:37.165129 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.164933 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:11:37.203392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203516 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203410 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-metrics-client-ca\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203516 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203435 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-telemeter-client-tls\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203516 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203506 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-secret-telemeter-client\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203646 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203678 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203644 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-serving-certs-ca-bundle\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203761 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203735 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbx87\" (UniqueName: \"kubernetes.io/projected/fcec410d-4f20-4660-a555-80d334ef15c7-kube-api-access-cbx87\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.203814 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.203772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-federate-client-tls\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304696 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbx87\" (UniqueName: \"kubernetes.io/projected/fcec410d-4f20-4660-a555-80d334ef15c7-kube-api-access-cbx87\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304732 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-federate-client-tls\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304753 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-metrics-client-ca\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304820 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-telemeter-client-tls\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304848 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-secret-telemeter-client\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.304965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.304953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-serving-certs-ca-bundle\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.305719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.305686 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-metrics-client-ca\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.305847 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.305801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-serving-certs-ca-bundle\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.305847 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.305801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fcec410d-4f20-4660-a555-80d334ef15c7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.308101 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.308078 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-secret-telemeter-client\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.308252 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.308231 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-federate-client-tls\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.308560 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.308539 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-telemeter-client-tls\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.308930 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.308903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/fcec410d-4f20-4660-a555-80d334ef15c7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.321737 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.321668 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbx87\" (UniqueName: \"kubernetes.io/projected/fcec410d-4f20-4660-a555-80d334ef15c7-kube-api-access-cbx87\") pod \"telemeter-client-6c687b8b78-svdjj\" (UID: \"fcec410d-4f20-4660-a555-80d334ef15c7\") " pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.385123 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.385091 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" Jul 15 13:11:37.644239 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.644203 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c687b8b78-svdjj"] Jul 15 13:11:37.645963 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:37.645931 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcec410d_4f20_4660_a555_80d334ef15c7.slice/crio-24bc4fdbf6eec52596458c9307689e3ea3617d839d7fe8ef5c2207725d9c6db2 WatchSource:0}: Error finding container 24bc4fdbf6eec52596458c9307689e3ea3617d839d7fe8ef5c2207725d9c6db2: Status 404 returned error can't find the container with id 24bc4fdbf6eec52596458c9307689e3ea3617d839d7fe8ef5c2207725d9c6db2 Jul 15 13:11:37.676582 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.676558 2564 generic.go:356] "Generic (PLEG): container finished" podID="f2aeca5e-acaf-415c-aa19-44ebf58cbd4d" containerID="bca9635e6e10495b8e686ce03f4cb696a0ac930492096f513b6bf496e16968de" exitCode=0 Jul 15 13:11:37.676911 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.676656 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2cwm" event={"ID":"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d","Type":"ContainerDied","Data":"bca9635e6e10495b8e686ce03f4cb696a0ac930492096f513b6bf496e16968de"} Jul 15 13:11:37.678074 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.678043 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" event={"ID":"fcec410d-4f20-4660-a555-80d334ef15c7","Type":"ContainerStarted","Data":"24bc4fdbf6eec52596458c9307689e3ea3617d839d7fe8ef5c2207725d9c6db2"} Jul 15 13:11:37.679331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.679304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" event={"ID":"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970","Type":"ContainerStarted","Data":"3f18566a7fdbac3d43e30ef16a46dea0d5adcd426530cda291ce69cb60025d10"} Jul 15 13:11:37.682059 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.682038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"5cfc890d08729665aa7efa8c35df26595dd52ae01f8041d868658bb7e68750f0"} Jul 15 13:11:37.682140 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.682088 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"c0abe9343fef350f766b5c031da8ae21f9e62e2a649a7cc2d567d5dc2341ee84"} Jul 15 13:11:37.682140 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.682115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"5ece2a25837661f52454d588011052c7c3a8d905985a2160f1e45ebad39af073"} Jul 15 13:11:37.684013 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.683992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"8b68dd5b0f68ab7a9c3b9bf545bf9d5f2ed63c4bc2a78bac5c5631fcd889ce05"} Jul 15 13:11:37.684109 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:37.684056 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"8f6c75f4c12718a599fdc6f3a06f1deb57ef88c5056b37bd07b571ca88071121"} Jul 15 13:11:38.126693 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.126612 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 13:11:38.131678 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.131652 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.134174 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.134146 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 15 13:11:38.134432 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.134404 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 15 13:11:38.134432 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.134412 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-7aaatgdpc7eae\"" Jul 15 13:11:38.134635 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.134460 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 15 13:11:38.134769 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.134750 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 15 13:11:38.135079 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.134932 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 15 13:11:38.135818 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.135797 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 15 13:11:38.136284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.136167 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 15 13:11:38.136284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.136173 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 15 13:11:38.136284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.136223 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 15 13:11:38.136539 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.136287 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 15 13:11:38.136539 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.136290 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 15 13:11:38.137027 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.136861 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4t4sq\"" Jul 15 13:11:38.138843 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.138718 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 15 13:11:38.140012 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.139988 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 15 13:11:38.142700 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.142680 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 15 13:11:38.151762 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.151683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 13:11:38.213404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213373 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213418 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213498 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213548 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213600 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcv2l\" (UniqueName: \"kubernetes.io/projected/e57f676a-b7e9-4617-b5da-7ddf415925c1-kube-api-access-gcv2l\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213665 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e57f676a-b7e9-4617-b5da-7ddf415925c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213966 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213966 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213847 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213966 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.213966 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213972 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.213998 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.214030 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.214054 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.214086 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e57f676a-b7e9-4617-b5da-7ddf415925c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.214112 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.214138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.214460 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.214168 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-config\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315522 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315522 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcv2l\" (UniqueName: \"kubernetes.io/projected/e57f676a-b7e9-4617-b5da-7ddf415925c1-kube-api-access-gcv2l\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315732 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315732 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e57f676a-b7e9-4617-b5da-7ddf415925c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315845 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315770 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315845 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315800 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315845 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315899 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315952 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.315999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.315985 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316012 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316038 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316075 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e57f676a-b7e9-4617-b5da-7ddf415925c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316104 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-config\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316198 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.316451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316272 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.317578 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.316645 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.317578 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.317060 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.317578 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.317296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.317578 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.317423 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.318029 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.318001 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.318817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.318228 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.318817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.318750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.319326 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.319238 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.319326 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.319268 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e57f676a-b7e9-4617-b5da-7ddf415925c1-config-out\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.320065 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.320021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e57f676a-b7e9-4617-b5da-7ddf415925c1-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.320167 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.320036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-web-config\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.320787 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.320741 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.320876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.320846 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.321398 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.321357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.321621 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.321588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e57f676a-b7e9-4617-b5da-7ddf415925c1-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.321708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.321688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.322285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.322262 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.322600 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.322580 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-config\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.322732 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.322622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e57f676a-b7e9-4617-b5da-7ddf415925c1-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.324445 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.324425 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcv2l\" (UniqueName: \"kubernetes.io/projected/e57f676a-b7e9-4617-b5da-7ddf415925c1-kube-api-access-gcv2l\") pod \"prometheus-k8s-0\" (UID: \"e57f676a-b7e9-4617-b5da-7ddf415925c1\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.445099 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.445012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:38.692712 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.692670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"c44fdbb97919d273a2b0403b7bb50766ecd60fe1674e302a876c7c50f4fc4340"} Jul 15 13:11:38.693108 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.692758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"724b8d5c789c6b3ee6c0f4859039356f4cd8443324a101d80ea6e2a7f290e4ce"} Jul 15 13:11:38.693108 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.692801 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"5713c858dc13a741ce010591a54a85867ba58af3b9b6124c10d3ba2fb0b08a22"} Jul 15 13:11:38.695006 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.694980 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2cwm" event={"ID":"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d","Type":"ContainerStarted","Data":"d681a00a7632fd2a1ad29e052e182d9d01376d0dfa431936fb58e8489feb7a71"} Jul 15 13:11:38.695172 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.695027 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-c2cwm" event={"ID":"f2aeca5e-acaf-415c-aa19-44ebf58cbd4d","Type":"ContainerStarted","Data":"adf62c45879e5a5127086e3a0ef4be76686d2fa54dd088e84e0984039f18f477"} Jul 15 13:11:38.716382 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.716331 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-c2cwm" podStartSLOduration=2.993252594 podStartE2EDuration="7.716310911s" totalImagesPullingTime="4.633462145s" totalInitContainerRuntime="89.596172ms" isStatefulPod=true podCreationTimestamp="2026-07-15 13:11:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:38.714775738 +0000 UTC m=+304.670230312" watchObservedRunningTime="2026-07-15 13:11:38.716310911 +0000 UTC m=+304.671765477" Jul 15 13:11:38.987246 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:38.987216 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 15 13:11:38.991617 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:11:38.991573 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57f676a_b7e9_4617_b5da_7ddf415925c1.slice/crio-bd90d28fb542a2aa61e47a97e7265dbd710200ce6cadffa3f8ef8ba7619da9b3 WatchSource:0}: Error finding container bd90d28fb542a2aa61e47a97e7265dbd710200ce6cadffa3f8ef8ba7619da9b3: Status 404 returned error can't find the container with id bd90d28fb542a2aa61e47a97e7265dbd710200ce6cadffa3f8ef8ba7619da9b3 Jul 15 13:11:39.701955 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.701922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"47432cfec8a9568de1ea72c937ddfd0946583bab571d8b1b79b392e4a76f5ec6"} Jul 15 13:11:39.702280 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.702037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:39.702280 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.702067 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"fc81b60ae7667640a66470057797b30f4c4023d44d23691f7631750f2e283344"} Jul 15 13:11:39.702280 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.702098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" event={"ID":"f6aec2de-f43e-4b1f-aed1-fc326495ce6e","Type":"ContainerStarted","Data":"36c9b7cf3e10da36f7ba07200039c8f25b6f056ec155074283d20a351af10a33"} Jul 15 13:11:39.704919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.704895 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" event={"ID":"1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970","Type":"ContainerStarted","Data":"06197a5bd6e7769fbb5a31c3dedcfa82ba7832f450ec861913158e9b3a5ebef5"} Jul 15 13:11:39.706314 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.706284 2564 generic.go:356] "Generic (PLEG): container finished" podID="e57f676a-b7e9-4617-b5da-7ddf415925c1" containerID="06cb35894be5eea445805dee7661ed1561ed4b14d128f551bf7f32fd4202e123" exitCode=0 Jul 15 13:11:39.706396 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.706327 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerDied","Data":"06cb35894be5eea445805dee7661ed1561ed4b14d128f551bf7f32fd4202e123"} Jul 15 13:11:39.706396 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.706370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"bd90d28fb542a2aa61e47a97e7265dbd710200ce6cadffa3f8ef8ba7619da9b3"} Jul 15 13:11:39.728693 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.728627 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" podStartSLOduration=3.258383813 podStartE2EDuration="6.728614541s" totalImagesPullingTime="3.470230728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:39.727330673 +0000 UTC m=+305.682785280" watchObservedRunningTime="2026-07-15 13:11:39.728614541 +0000 UTC m=+305.684069123" Jul 15 13:11:39.774358 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:39.774315 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" podStartSLOduration=2.109458398 podStartE2EDuration="3.77429724s" totalImagesPullingTime="1.664838842s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:39.772430639 +0000 UTC m=+305.727885203" watchObservedRunningTime="2026-07-15 13:11:39.77429724 +0000 UTC m=+305.729751866" Jul 15 13:11:40.712662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:40.712594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" event={"ID":"fcec410d-4f20-4660-a555-80d334ef15c7","Type":"ContainerStarted","Data":"aef4d81652c5927c4147f1690be07ebe3f76e40c1546512a1664492577d68472"} Jul 15 13:11:40.713217 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:40.712670 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" event={"ID":"fcec410d-4f20-4660-a555-80d334ef15c7","Type":"ContainerStarted","Data":"7047eab9c91b513acea0e079ec220e3299b6b5f3471e5f8c8c8bc3fe1eb1af1a"} Jul 15 13:11:40.713217 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:40.712700 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" event={"ID":"fcec410d-4f20-4660-a555-80d334ef15c7","Type":"ContainerStarted","Data":"9c6490b17aa9615109d6a16cefc3f7ece48c8a157a5efceaea90700f0d1f27cd"} Jul 15 13:11:40.745251 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:40.745204 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c687b8b78-svdjj" podStartSLOduration=1.760022836 podStartE2EDuration="3.745186503s" totalImagesPullingTime="1.985163667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:40.743279635 +0000 UTC m=+306.698734198" watchObservedRunningTime="2026-07-15 13:11:40.745186503 +0000 UTC m=+306.700641067" Jul 15 13:11:41.721296 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:41.721256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d5252322-2e18-42fb-b72b-392ef40ceacc","Type":"ContainerStarted","Data":"5dbea1ed3efc757deaa5b8c8e714b339761dd707760f5ff79209b98c360b3fee"} Jul 15 13:11:41.749811 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:41.749757 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.546029697 podStartE2EDuration="8.749736692s" totalImagesPullingTime="6.185409467s" totalInitContainerRuntime="18.297528ms" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:33 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:41.749104431 +0000 UTC m=+307.704558995" watchObservedRunningTime="2026-07-15 13:11:41.749736692 +0000 UTC m=+307.705191257" Jul 15 13:11:43.734154 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:43.734125 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"8d8d09ff7ee00e27992e500a688166968e346dd496125ac0d1c93e4cfc7c9d23"} Jul 15 13:11:43.734452 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:43.734186 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"36f93ae39c28225d2c25f53e1af47d1f27f9ad37fcf1c24d41feda4ca2b91ce3"} Jul 15 13:11:43.734452 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:43.734213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"9f0adaeb1fc40fc7a118176010b3c09fe3ad42933aca3aea21e11eed6a07459c"} Jul 15 13:11:44.741588 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:44.741550 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"77941601867ae38bc4178adc5887190805eefa178cf5f49256b9b0f5418c71bf"} Jul 15 13:11:44.742040 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:44.741621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"455dcddc7e3bc65ab795f28443182a77d364cae36e0fe837f02158eaaf26d6f1"} Jul 15 13:11:44.742040 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:44.741660 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e57f676a-b7e9-4617-b5da-7ddf415925c1","Type":"ContainerStarted","Data":"5dcb94ff38f3ce604ce808fb96287885550be08d1ddfa651b0d7903483b534f6"} Jul 15 13:11:44.783233 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:44.782912 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.151316229 podStartE2EDuration="6.78289343s" totalImagesPullingTime="3.602424725s" totalInitContainerRuntime="29.152476ms" isStatefulPod=false podCreationTimestamp="2026-07-15 13:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:11:44.78127482 +0000 UTC m=+310.736729382" watchObservedRunningTime="2026-07-15 13:11:44.78289343 +0000 UTC m=+310.738347996" Jul 15 13:11:45.718351 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:45.718316 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6897b47b98-8v9z9" Jul 15 13:11:48.445335 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:48.445297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:11:52.699393 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:52.699363 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcf7bcdbf-lx4kt"] Jul 15 13:11:56.473579 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:56.473547 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:11:56.473935 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:11:56.473605 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:12:16.478793 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:16.478732 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:12:16.484946 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:16.484914 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-78999c7f87-qbnww" Jul 15 13:12:17.720631 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:17.720551 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7bcf7bcdbf-lx4kt" podUID="a5138255-6be6-4595-a07e-636121c55035" containerName="console" containerID="cri-o://07223b2de88370ee503c65f7bd65d8f5c3adce2970776195cf90114e73d7263f" gracePeriod=15 Jul 15 13:12:17.890178 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:17.890146 2564 generic.go:356] "Generic (PLEG): container finished" podID="a5138255-6be6-4595-a07e-636121c55035" containerID="07223b2de88370ee503c65f7bd65d8f5c3adce2970776195cf90114e73d7263f" exitCode=0 Jul 15 13:12:17.890349 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:17.890213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcf7bcdbf-lx4kt" event={"ID":"a5138255-6be6-4595-a07e-636121c55035","Type":"ContainerDied","Data":"07223b2de88370ee503c65f7bd65d8f5c3adce2970776195cf90114e73d7263f"} Jul 15 13:12:17.982799 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:17.982411 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:12:18.074000 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.073967 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-oauth-config\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074000 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074005 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-trusted-ca-bundle\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074029 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-serving-cert\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074075 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-service-ca\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074108 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-oauth-serving-cert\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074146 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-console-config\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074229 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mq94j\" (UniqueName: \"kubernetes.io/projected/a5138255-6be6-4595-a07e-636121c55035-kube-api-access-mq94j\") pod \"a5138255-6be6-4595-a07e-636121c55035\" (UID: \"a5138255-6be6-4595-a07e-636121c55035\") " Jul 15 13:12:18.074543 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074405 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:12:18.074596 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074542 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-trusted-ca-bundle\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.074596 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074540 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:12:18.074596 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074549 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-service-ca" (OuterVolumeSpecName: "service-ca") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:12:18.074596 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.074553 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-console-config" (OuterVolumeSpecName: "console-config") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:12:18.076384 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.076359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:12:18.076384 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.076378 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:12:18.076590 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.076426 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5138255-6be6-4595-a07e-636121c55035-kube-api-access-mq94j" (OuterVolumeSpecName: "kube-api-access-mq94j") pod "a5138255-6be6-4595-a07e-636121c55035" (UID: "a5138255-6be6-4595-a07e-636121c55035"). InnerVolumeSpecName "kube-api-access-mq94j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:12:18.175614 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.175581 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-oauth-serving-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.175614 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.175606 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-console-config\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.175614 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.175614 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq94j\" (UniqueName: \"kubernetes.io/projected/a5138255-6be6-4595-a07e-636121c55035-kube-api-access-mq94j\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.175823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.175625 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-oauth-config\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.175823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.175635 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a5138255-6be6-4595-a07e-636121c55035-console-serving-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.175823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.175643 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a5138255-6be6-4595-a07e-636121c55035-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:12:18.896070 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.896025 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7bcf7bcdbf-lx4kt" Jul 15 13:12:18.896489 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.896029 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7bcf7bcdbf-lx4kt" event={"ID":"a5138255-6be6-4595-a07e-636121c55035","Type":"ContainerDied","Data":"5a73159e7a9bdcf3bf797a86c3d7f14d4b72cbd83b7409f1939d47ebfaa3790a"} Jul 15 13:12:18.896489 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.896165 2564 scope.go:117] "RemoveContainer" containerID="07223b2de88370ee503c65f7bd65d8f5c3adce2970776195cf90114e73d7263f" Jul 15 13:12:18.913030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.913006 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7bcf7bcdbf-lx4kt"] Jul 15 13:12:18.918989 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:18.918936 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7bcf7bcdbf-lx4kt"] Jul 15 13:12:20.657062 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:20.657033 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5138255-6be6-4595-a07e-636121c55035" path="/var/lib/kubelet/pods/a5138255-6be6-4595-a07e-636121c55035/volumes" Jul 15 13:12:38.445431 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:38.445396 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:12:38.461437 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:38.461412 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:12:38.995789 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:12:38.995764 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 15 13:14:47.518131 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.518096 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q"] Jul 15 13:14:47.518566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.518454 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5138255-6be6-4595-a07e-636121c55035" containerName="console" Jul 15 13:14:47.518566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.518489 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5138255-6be6-4595-a07e-636121c55035" containerName="console" Jul 15 13:14:47.518566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.518558 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5138255-6be6-4595-a07e-636121c55035" containerName="console" Jul 15 13:14:47.521761 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.521730 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.527696 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.527671 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-snv8d\"" Jul 15 13:14:47.527854 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.527792 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 15 13:14:47.527854 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.527792 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 15 13:14:47.528309 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.528028 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 15 13:14:47.528396 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.528379 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 15 13:14:47.568635 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.568596 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q"] Jul 15 13:14:47.627424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.627375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-manifests\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.627617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.627490 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4znvt\" (UniqueName: \"kubernetes.io/projected/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-kube-api-access-4znvt\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.627617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.627545 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.627617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.627578 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-tmp\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.627617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.627594 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-webhook-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.728857 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.728813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.729028 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.728987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-tmp\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.729084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.729032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-webhook-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.729162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.729145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-manifests\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.729284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.729256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4znvt\" (UniqueName: \"kubernetes.io/projected/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-kube-api-access-4znvt\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.729510 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.729456 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-manifests\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.731490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.731433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-webhook-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.731676 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.731653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-tmp\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.731880 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.731862 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.737258 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.737237 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4znvt\" (UniqueName: \"kubernetes.io/projected/8c24eee3-8564-4b2c-8c87-a59c70fc1da4-kube-api-access-4znvt\") pod \"opendatahub-operator-controller-manager-869c48857d-rfq5q\" (UID: \"8c24eee3-8564-4b2c-8c87-a59c70fc1da4\") " pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.832380 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.832340 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:47.979577 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:47.979552 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q"] Jul 15 13:14:47.981884 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:14:47.981850 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c24eee3_8564_4b2c_8c87_a59c70fc1da4.slice/crio-a9a32e64bceb559a548c1d1f5a3d976ec725fedc685dbf4885772ffabb742dc1 WatchSource:0}: Error finding container a9a32e64bceb559a548c1d1f5a3d976ec725fedc685dbf4885772ffabb742dc1: Status 404 returned error can't find the container with id a9a32e64bceb559a548c1d1f5a3d976ec725fedc685dbf4885772ffabb742dc1 Jul 15 13:14:48.519223 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:48.519184 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" event={"ID":"8c24eee3-8564-4b2c-8c87-a59c70fc1da4","Type":"ContainerStarted","Data":"a9a32e64bceb559a548c1d1f5a3d976ec725fedc685dbf4885772ffabb742dc1"} Jul 15 13:14:51.539395 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:51.539359 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" event={"ID":"8c24eee3-8564-4b2c-8c87-a59c70fc1da4","Type":"ContainerStarted","Data":"4eaae4fad85adec18fd04eb061e374d7f283dfe4a1fbd7efdc349dd740aefb33"} Jul 15 13:14:53.550271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:53.550241 2564 generic.go:356] "Generic (PLEG): container finished" podID="8c24eee3-8564-4b2c-8c87-a59c70fc1da4" containerID="4eaae4fad85adec18fd04eb061e374d7f283dfe4a1fbd7efdc349dd740aefb33" exitCode=0 Jul 15 13:14:53.550644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:53.550312 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" event={"ID":"8c24eee3-8564-4b2c-8c87-a59c70fc1da4","Type":"ContainerDied","Data":"4eaae4fad85adec18fd04eb061e374d7f283dfe4a1fbd7efdc349dd740aefb33"} Jul 15 13:14:54.557729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:54.557696 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" event={"ID":"8c24eee3-8564-4b2c-8c87-a59c70fc1da4","Type":"ContainerStarted","Data":"5f6d7b4a658153ebda312fa9cc4c41c1b69ce80d202748950bdf162a9f06609a"} Jul 15 13:14:54.558142 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:54.557926 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:14:54.584384 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:14:54.584337 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" podStartSLOduration=2.53928488 podStartE2EDuration="7.584323385s" totalImagesPullingTime="3.949360143s" totalInitContainerRuntime="1.095678362s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:14:47 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:14:54.582692554 +0000 UTC m=+500.538147117" watchObservedRunningTime="2026-07-15 13:14:54.584323385 +0000 UTC m=+500.539777947" Jul 15 13:15:05.564045 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:05.564014 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-869c48857d-rfq5q" Jul 15 13:15:07.348998 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.348965 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87"] Jul 15 13:15:07.355752 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.355728 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.359288 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.359264 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 15 13:15:07.359430 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.359264 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 15 13:15:07.359430 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.359301 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zxppv\"" Jul 15 13:15:07.359430 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.359265 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 15 13:15:07.359637 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.359267 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 15 13:15:07.359637 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.359284 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 15 13:15:07.369491 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.369452 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87"] Jul 15 13:15:07.522949 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.522903 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-metrics-cert\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.522949 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.522958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-cert\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.523163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.523041 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-manager-config\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.523163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.523073 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd59v\" (UniqueName: \"kubernetes.io/projected/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-kube-api-access-rd59v\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.624360 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.624274 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-cert\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.624557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.624442 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-manager-config\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.624557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.624527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd59v\" (UniqueName: \"kubernetes.io/projected/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-kube-api-access-rd59v\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.624650 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.624574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-metrics-cert\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.625036 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.625016 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-manager-config\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.626957 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.626937 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-metrics-cert\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.626957 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.626946 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-cert\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.633820 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.633799 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd59v\" (UniqueName: \"kubernetes.io/projected/0ae2fb56-f03c-48c4-8943-62cfb5dad3ec-kube-api-access-rd59v\") pod \"lws-controller-manager-8ddc6f8f9-l7q87\" (UID: \"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec\") " pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.665636 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.665602 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:07.797437 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:07.797412 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87"] Jul 15 13:15:07.799824 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:15:07.799797 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae2fb56_f03c_48c4_8943_62cfb5dad3ec.slice/crio-d8f64fa370b0a579abf2e409f87e2cd448dedbac5182158c0ff64952613ec119 WatchSource:0}: Error finding container d8f64fa370b0a579abf2e409f87e2cd448dedbac5182158c0ff64952613ec119: Status 404 returned error can't find the container with id d8f64fa370b0a579abf2e409f87e2cd448dedbac5182158c0ff64952613ec119 Jul 15 13:15:08.623525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:08.623464 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" event={"ID":"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec","Type":"ContainerStarted","Data":"d8f64fa370b0a579abf2e409f87e2cd448dedbac5182158c0ff64952613ec119"} Jul 15 13:15:10.636051 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:10.636009 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" event={"ID":"0ae2fb56-f03c-48c4-8943-62cfb5dad3ec","Type":"ContainerStarted","Data":"8de32cf3a6725560fd7852030f00d3cf06c04528ff4869ab28936815c36945fa"} Jul 15 13:15:10.636549 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:10.636140 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:15:10.660372 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:10.660305 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" podStartSLOduration=1.4650363020000001 podStartE2EDuration="3.660282104s" totalImagesPullingTime="2.195245802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:10.653705008 +0000 UTC m=+516.609159602" watchObservedRunningTime="2026-07-15 13:15:10.660282104 +0000 UTC m=+516.615736665" Jul 15 13:15:11.363617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.363579 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw"] Jul 15 13:15:11.367832 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.367758 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.371226 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.371201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 15 13:15:11.371919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.371892 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-8m4zb\"" Jul 15 13:15:11.371919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.371919 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 15 13:15:11.381415 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.381390 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw"] Jul 15 13:15:11.465846 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.465815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfq8r\" (UniqueName: \"kubernetes.io/projected/799fe46c-9193-4755-858d-0d60152b4fc1-kube-api-access-tfq8r\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.466030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.465865 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799fe46c-9193-4755-858d-0d60152b4fc1-tmp\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.466030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.465939 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799fe46c-9193-4755-858d-0d60152b4fc1-tls-certs\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.567336 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.567299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfq8r\" (UniqueName: \"kubernetes.io/projected/799fe46c-9193-4755-858d-0d60152b4fc1-kube-api-access-tfq8r\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.567537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.567346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799fe46c-9193-4755-858d-0d60152b4fc1-tmp\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.567537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.567367 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799fe46c-9193-4755-858d-0d60152b4fc1-tls-certs\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.569673 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.569648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/799fe46c-9193-4755-858d-0d60152b4fc1-tmp\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.570003 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.569981 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/799fe46c-9193-4755-858d-0d60152b4fc1-tls-certs\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.576589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.576565 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfq8r\" (UniqueName: \"kubernetes.io/projected/799fe46c-9193-4755-858d-0d60152b4fc1-kube-api-access-tfq8r\") pod \"kube-auth-proxy-6b6cfc997-5k7bw\" (UID: \"799fe46c-9193-4755-858d-0d60152b4fc1\") " pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.680754 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.680666 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" Jul 15 13:15:11.831455 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:11.831422 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw"] Jul 15 13:15:11.833541 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:15:11.833512 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799fe46c_9193_4755_858d_0d60152b4fc1.slice/crio-6dec2e250dc6cfa55be95c42198c1b7dedd3d294892e9c33c6be93833560ad87 WatchSource:0}: Error finding container 6dec2e250dc6cfa55be95c42198c1b7dedd3d294892e9c33c6be93833560ad87: Status 404 returned error can't find the container with id 6dec2e250dc6cfa55be95c42198c1b7dedd3d294892e9c33c6be93833560ad87 Jul 15 13:15:12.648964 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:12.648926 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" event={"ID":"799fe46c-9193-4755-858d-0d60152b4fc1","Type":"ContainerStarted","Data":"6dec2e250dc6cfa55be95c42198c1b7dedd3d294892e9c33c6be93833560ad87"} Jul 15 13:15:15.666828 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:15.666735 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" event={"ID":"799fe46c-9193-4755-858d-0d60152b4fc1","Type":"ContainerStarted","Data":"d57d0b115e8991671c5e2e1d56f2839ccd783c0b4cf77e7d71b25e98516c9e00"} Jul 15 13:15:15.684312 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:15.684239 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b6cfc997-5k7bw" podStartSLOduration=1.142087109 podStartE2EDuration="4.684222282s" totalImagesPullingTime="3.542135173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:15:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:15:15.683322008 +0000 UTC m=+521.638776572" watchObservedRunningTime="2026-07-15 13:15:15.684222282 +0000 UTC m=+521.639676847" Jul 15 13:15:21.644662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:15:21.644579 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8ddc6f8f9-l7q87" Jul 15 13:16:34.518988 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:34.518956 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:16:34.519721 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:34.519242 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:16:49.378454 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.378419 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59646cffc8-tk9hx"] Jul 15 13:16:49.381922 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.381903 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.384294 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.384272 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 15 13:16:49.384432 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.384414 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 15 13:16:49.384517 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.384440 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 15 13:16:49.385182 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.385165 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 15 13:16:49.385182 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.385178 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-f5tck\"" Jul 15 13:16:49.385314 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.385184 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 15 13:16:49.385314 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.385230 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 15 13:16:49.385314 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.385165 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 15 13:16:49.391882 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.391864 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 15 13:16:49.395546 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.395525 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59646cffc8-tk9hx"] Jul 15 13:16:49.428757 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-config\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.428878 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428765 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-serving-cert\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.428878 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85jqf\" (UniqueName: \"kubernetes.io/projected/4d89908d-52a0-4b9c-af0a-e91980b349e4-kube-api-access-85jqf\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.428878 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-oauth-config\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.428878 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-oauth-serving-cert\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.429072 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428891 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-trusted-ca-bundle\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.429072 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.428958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-service-ca\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529544 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529513 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-trusted-ca-bundle\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-service-ca\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-config\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-serving-cert\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85jqf\" (UniqueName: \"kubernetes.io/projected/4d89908d-52a0-4b9c-af0a-e91980b349e4-kube-api-access-85jqf\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-oauth-config\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.529841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.529791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-oauth-serving-cert\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.530525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.530495 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-service-ca\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.530666 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.530648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-config\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.530734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.530688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-oauth-serving-cert\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.530792 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.530747 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4d89908d-52a0-4b9c-af0a-e91980b349e4-trusted-ca-bundle\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.532191 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.532169 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-oauth-config\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.532294 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.532261 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4d89908d-52a0-4b9c-af0a-e91980b349e4-console-serving-cert\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.548980 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.548958 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85jqf\" (UniqueName: \"kubernetes.io/projected/4d89908d-52a0-4b9c-af0a-e91980b349e4-kube-api-access-85jqf\") pod \"console-59646cffc8-tk9hx\" (UID: \"4d89908d-52a0-4b9c-af0a-e91980b349e4\") " pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.691786 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.691714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:49.824492 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:49.824386 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59646cffc8-tk9hx"] Jul 15 13:16:49.826793 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:16:49.826764 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d89908d_52a0_4b9c_af0a_e91980b349e4.slice/crio-1bdf82a5fa079e954009987a6edd7afd79e1970e5c42ccf743663f6257aa4410 WatchSource:0}: Error finding container 1bdf82a5fa079e954009987a6edd7afd79e1970e5c42ccf743663f6257aa4410: Status 404 returned error can't find the container with id 1bdf82a5fa079e954009987a6edd7afd79e1970e5c42ccf743663f6257aa4410 Jul 15 13:16:50.115951 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:50.115920 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59646cffc8-tk9hx" event={"ID":"4d89908d-52a0-4b9c-af0a-e91980b349e4","Type":"ContainerStarted","Data":"d97c17b0aa1cf7ef9b426fd2a14a414cf9dedb409a1496c8b8099483934f41fa"} Jul 15 13:16:50.116110 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:50.115972 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59646cffc8-tk9hx" event={"ID":"4d89908d-52a0-4b9c-af0a-e91980b349e4","Type":"ContainerStarted","Data":"1bdf82a5fa079e954009987a6edd7afd79e1970e5c42ccf743663f6257aa4410"} Jul 15 13:16:50.144837 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:50.144653 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59646cffc8-tk9hx" podStartSLOduration=1.144634344 podStartE2EDuration="1.144634344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:50.14329181 +0000 UTC m=+616.098746373" watchObservedRunningTime="2026-07-15 13:16:50.144634344 +0000 UTC m=+616.100088908" Jul 15 13:16:53.536390 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.536112 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp"] Jul 15 13:16:53.539986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.539967 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:16:53.542354 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.542331 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 15 13:16:53.542435 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.542358 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 15 13:16:53.543198 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.543179 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-5vgcj\"" Jul 15 13:16:53.549484 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.549441 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp"] Jul 15 13:16:53.567980 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.567957 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlhxc\" (UniqueName: \"kubernetes.io/projected/c8d90403-666b-4c64-9ebe-7ff8419c4b44-kube-api-access-zlhxc\") pod \"authorino-operator-75cc8d5cb7-f2tcp\" (UID: \"c8d90403-666b-4c64-9ebe-7ff8419c4b44\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:16:53.669197 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.669159 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlhxc\" (UniqueName: \"kubernetes.io/projected/c8d90403-666b-4c64-9ebe-7ff8419c4b44-kube-api-access-zlhxc\") pod \"authorino-operator-75cc8d5cb7-f2tcp\" (UID: \"c8d90403-666b-4c64-9ebe-7ff8419c4b44\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:16:53.677292 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.677272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlhxc\" (UniqueName: \"kubernetes.io/projected/c8d90403-666b-4c64-9ebe-7ff8419c4b44-kube-api-access-zlhxc\") pod \"authorino-operator-75cc8d5cb7-f2tcp\" (UID: \"c8d90403-666b-4c64-9ebe-7ff8419c4b44\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:16:53.851084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.851017 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:16:53.997846 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:53.997820 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp"] Jul 15 13:16:54.000318 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:16:54.000293 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d90403_666b_4c64_9ebe_7ff8419c4b44.slice/crio-ce22ad32506e32f8f478634073c077ed096f4e68fdbce2864f8a56e0403b6c3a WatchSource:0}: Error finding container ce22ad32506e32f8f478634073c077ed096f4e68fdbce2864f8a56e0403b6c3a: Status 404 returned error can't find the container with id ce22ad32506e32f8f478634073c077ed096f4e68fdbce2864f8a56e0403b6c3a Jul 15 13:16:54.002456 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:54.002440 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:16:54.135513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:54.135411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" event={"ID":"c8d90403-666b-4c64-9ebe-7ff8419c4b44","Type":"ContainerStarted","Data":"ce22ad32506e32f8f478634073c077ed096f4e68fdbce2864f8a56e0403b6c3a"} Jul 15 13:16:57.155245 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:57.155209 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" event={"ID":"c8d90403-666b-4c64-9ebe-7ff8419c4b44","Type":"ContainerStarted","Data":"2f7f545a7f421a0ee35700f46c26119a50c3e85c128f1c9537c5ddef87b0e3db"} Jul 15 13:16:57.155612 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:57.155334 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:16:57.195585 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:57.195539 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" podStartSLOduration=1.9559088610000002 podStartE2EDuration="4.19552551s" totalImagesPullingTime="2.239616649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:16:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:16:57.194798925 +0000 UTC m=+623.150253489" watchObservedRunningTime="2026-07-15 13:16:57.19552551 +0000 UTC m=+623.150980072" Jul 15 13:16:59.691932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:59.691902 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:59.692300 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:59.691966 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:16:59.697057 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:16:59.697032 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:17:00.173633 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:17:00.173608 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59646cffc8-tk9hx" Jul 15 13:17:08.162315 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:17:08.162283 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-f2tcp" Jul 15 13:18:00.832002 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.831973 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-5s7zp"] Jul 15 13:18:00.834421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.834405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:00.837185 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.837164 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-k5snx\"" Jul 15 13:18:00.845959 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.845937 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5s7zp"] Jul 15 13:18:00.960335 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.960302 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbj4\" (UniqueName: \"kubernetes.io/projected/1589b401-68b3-443a-b277-e2f0e8aec0cd-kube-api-access-mcbj4\") pod \"authorino-58cd955675-5s7zp\" (UID: \"1589b401-68b3-443a-b277-e2f0e8aec0cd\") " pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:00.970485 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.970444 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-hkb7w"] Jul 15 13:18:00.973012 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.972997 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:00.980548 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:00.980525 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hkb7w"] Jul 15 13:18:01.061684 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.061660 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbj4\" (UniqueName: \"kubernetes.io/projected/1589b401-68b3-443a-b277-e2f0e8aec0cd-kube-api-access-mcbj4\") pod \"authorino-58cd955675-5s7zp\" (UID: \"1589b401-68b3-443a-b277-e2f0e8aec0cd\") " pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:01.061823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.061739 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4t4\" (UniqueName: \"kubernetes.io/projected/48115549-8eaf-4212-bfff-407de9d8dbd6-kube-api-access-ts4t4\") pod \"authorino-75d556bd77-hkb7w\" (UID: \"48115549-8eaf-4212-bfff-407de9d8dbd6\") " pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:01.071763 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.071740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbj4\" (UniqueName: \"kubernetes.io/projected/1589b401-68b3-443a-b277-e2f0e8aec0cd-kube-api-access-mcbj4\") pod \"authorino-58cd955675-5s7zp\" (UID: \"1589b401-68b3-443a-b277-e2f0e8aec0cd\") " pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:01.154817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.154759 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:01.162986 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.162966 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4t4\" (UniqueName: \"kubernetes.io/projected/48115549-8eaf-4212-bfff-407de9d8dbd6-kube-api-access-ts4t4\") pod \"authorino-75d556bd77-hkb7w\" (UID: \"48115549-8eaf-4212-bfff-407de9d8dbd6\") " pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:01.175347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.175327 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4t4\" (UniqueName: \"kubernetes.io/projected/48115549-8eaf-4212-bfff-407de9d8dbd6-kube-api-access-ts4t4\") pod \"authorino-75d556bd77-hkb7w\" (UID: \"48115549-8eaf-4212-bfff-407de9d8dbd6\") " pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:01.282974 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.282874 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:01.284648 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.284617 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-5s7zp"] Jul 15 13:18:01.286116 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:18:01.286087 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1589b401_68b3_443a_b277_e2f0e8aec0cd.slice/crio-8bdf3b2d985722bec04dec3135aa5270e1dad6b9baf6b2a23a1dcb6f8b46aeca WatchSource:0}: Error finding container 8bdf3b2d985722bec04dec3135aa5270e1dad6b9baf6b2a23a1dcb6f8b46aeca: Status 404 returned error can't find the container with id 8bdf3b2d985722bec04dec3135aa5270e1dad6b9baf6b2a23a1dcb6f8b46aeca Jul 15 13:18:01.412011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.411986 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hkb7w"] Jul 15 13:18:01.413591 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:18:01.413563 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48115549_8eaf_4212_bfff_407de9d8dbd6.slice/crio-74c62890b409ec0cb5e3ca501950a555e101ba5903e677a01ec3b630c841cc20 WatchSource:0}: Error finding container 74c62890b409ec0cb5e3ca501950a555e101ba5903e677a01ec3b630c841cc20: Status 404 returned error can't find the container with id 74c62890b409ec0cb5e3ca501950a555e101ba5903e677a01ec3b630c841cc20 Jul 15 13:18:01.456554 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.456528 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hkb7w" event={"ID":"48115549-8eaf-4212-bfff-407de9d8dbd6","Type":"ContainerStarted","Data":"74c62890b409ec0cb5e3ca501950a555e101ba5903e677a01ec3b630c841cc20"} Jul 15 13:18:01.457584 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:01.457557 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5s7zp" event={"ID":"1589b401-68b3-443a-b277-e2f0e8aec0cd","Type":"ContainerStarted","Data":"8bdf3b2d985722bec04dec3135aa5270e1dad6b9baf6b2a23a1dcb6f8b46aeca"} Jul 15 13:18:06.487616 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:06.487575 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hkb7w" event={"ID":"48115549-8eaf-4212-bfff-407de9d8dbd6","Type":"ContainerStarted","Data":"ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4"} Jul 15 13:18:06.488978 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:06.488952 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5s7zp" event={"ID":"1589b401-68b3-443a-b277-e2f0e8aec0cd","Type":"ContainerStarted","Data":"0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5"} Jul 15 13:18:06.504733 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:06.504688 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-hkb7w" podStartSLOduration=2.095989764 podStartE2EDuration="6.504674455s" totalImagesPullingTime="4.408684691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:18:06.502265026 +0000 UTC m=+692.457719599" watchObservedRunningTime="2026-07-15 13:18:06.504674455 +0000 UTC m=+692.460129021" Jul 15 13:18:06.517655 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:06.517606 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-5s7zp" podStartSLOduration=1.969327121 podStartE2EDuration="6.517590052s" totalImagesPullingTime="4.548262931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:18:06.516777757 +0000 UTC m=+692.472232320" watchObservedRunningTime="2026-07-15 13:18:06.517590052 +0000 UTC m=+692.473044615" Jul 15 13:18:06.574509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:06.574482 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5s7zp"] Jul 15 13:18:08.498235 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:08.498191 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-5s7zp" podUID="1589b401-68b3-443a-b277-e2f0e8aec0cd" containerName="authorino" containerID="cri-o://0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5" gracePeriod=30 Jul 15 13:18:08.745533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:08.745510 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:08.838808 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:08.838779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcbj4\" (UniqueName: \"kubernetes.io/projected/1589b401-68b3-443a-b277-e2f0e8aec0cd-kube-api-access-mcbj4\") pod \"1589b401-68b3-443a-b277-e2f0e8aec0cd\" (UID: \"1589b401-68b3-443a-b277-e2f0e8aec0cd\") " Jul 15 13:18:08.840827 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:08.840806 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1589b401-68b3-443a-b277-e2f0e8aec0cd-kube-api-access-mcbj4" (OuterVolumeSpecName: "kube-api-access-mcbj4") pod "1589b401-68b3-443a-b277-e2f0e8aec0cd" (UID: "1589b401-68b3-443a-b277-e2f0e8aec0cd"). InnerVolumeSpecName "kube-api-access-mcbj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:18:08.939745 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:08.939722 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcbj4\" (UniqueName: \"kubernetes.io/projected/1589b401-68b3-443a-b277-e2f0e8aec0cd-kube-api-access-mcbj4\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:18:09.504766 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.504730 2564 generic.go:356] "Generic (PLEG): container finished" podID="1589b401-68b3-443a-b277-e2f0e8aec0cd" containerID="0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5" exitCode=0 Jul 15 13:18:09.505230 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.504774 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-5s7zp" Jul 15 13:18:09.505230 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.504792 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5s7zp" event={"ID":"1589b401-68b3-443a-b277-e2f0e8aec0cd","Type":"ContainerDied","Data":"0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5"} Jul 15 13:18:09.505230 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.504830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-5s7zp" event={"ID":"1589b401-68b3-443a-b277-e2f0e8aec0cd","Type":"ContainerDied","Data":"8bdf3b2d985722bec04dec3135aa5270e1dad6b9baf6b2a23a1dcb6f8b46aeca"} Jul 15 13:18:09.505230 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.504853 2564 scope.go:117] "RemoveContainer" containerID="0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5" Jul 15 13:18:09.513197 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.513182 2564 scope.go:117] "RemoveContainer" containerID="0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5" Jul 15 13:18:09.513478 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:18:09.513448 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5\": container with ID starting with 0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5 not found: ID does not exist" containerID="0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5" Jul 15 13:18:09.513530 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.513491 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5"} err="failed to get container status \"0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5\": rpc error: code = NotFound desc = could not find container \"0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5\": container with ID starting with 0e22c4fcfd894ca3b8b9cc28e9d6807bb18f6816c7c55b3b7eed6cd4fdb7b2e5 not found: ID does not exist" Jul 15 13:18:09.525299 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.525278 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-5s7zp"] Jul 15 13:18:09.529138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:09.529118 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-5s7zp"] Jul 15 13:18:10.657267 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:10.657239 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1589b401-68b3-443a-b277-e2f0e8aec0cd" path="/var/lib/kubelet/pods/1589b401-68b3-443a-b277-e2f0e8aec0cd/volumes" Jul 15 13:18:31.094343 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.094262 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5w66"] Jul 15 13:18:31.094734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.094712 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1589b401-68b3-443a-b277-e2f0e8aec0cd" containerName="authorino" Jul 15 13:18:31.094734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.094725 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1589b401-68b3-443a-b277-e2f0e8aec0cd" containerName="authorino" Jul 15 13:18:31.094819 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.094807 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1589b401-68b3-443a-b277-e2f0e8aec0cd" containerName="authorino" Jul 15 13:18:31.096936 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.096920 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:31.109938 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.109919 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5w66"] Jul 15 13:18:31.122852 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.122827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pk5ww\" (UniqueName: \"kubernetes.io/projected/0f96334a-483f-47f1-b2cc-ac131282c674-kube-api-access-pk5ww\") pod \"authorino-5d4b67c446-s5w66\" (UID: \"0f96334a-483f-47f1-b2cc-ac131282c674\") " pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:31.223785 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.223751 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pk5ww\" (UniqueName: \"kubernetes.io/projected/0f96334a-483f-47f1-b2cc-ac131282c674-kube-api-access-pk5ww\") pod \"authorino-5d4b67c446-s5w66\" (UID: \"0f96334a-483f-47f1-b2cc-ac131282c674\") " pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:31.231857 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.231836 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pk5ww\" (UniqueName: \"kubernetes.io/projected/0f96334a-483f-47f1-b2cc-ac131282c674-kube-api-access-pk5ww\") pod \"authorino-5d4b67c446-s5w66\" (UID: \"0f96334a-483f-47f1-b2cc-ac131282c674\") " pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:31.332203 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.332174 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5w66"] Jul 15 13:18:31.332412 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.332402 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:31.356404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.356351 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-569544f5cc-hwm7x"] Jul 15 13:18:31.358868 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.358849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:31.368746 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.368723 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-569544f5cc-hwm7x"] Jul 15 13:18:31.425370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.425324 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6p5\" (UniqueName: \"kubernetes.io/projected/022c8462-fceb-4a5b-8bf4-0ca55eefe7be-kube-api-access-dh6p5\") pod \"authorino-569544f5cc-hwm7x\" (UID: \"022c8462-fceb-4a5b-8bf4-0ca55eefe7be\") " pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:31.463639 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.463616 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5w66"] Jul 15 13:18:31.465964 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:18:31.465930 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f96334a_483f_47f1_b2cc_ac131282c674.slice/crio-82555ccdf7489ec42b3be32ca55ab4b2d1e38d024303057a5bde9c73fb522ad7 WatchSource:0}: Error finding container 82555ccdf7489ec42b3be32ca55ab4b2d1e38d024303057a5bde9c73fb522ad7: Status 404 returned error can't find the container with id 82555ccdf7489ec42b3be32ca55ab4b2d1e38d024303057a5bde9c73fb522ad7 Jul 15 13:18:31.468531 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.468511 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-569544f5cc-hwm7x"] Jul 15 13:18:31.468777 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:18:31.468761 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-dh6p5], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-569544f5cc-hwm7x" podUID="022c8462-fceb-4a5b-8bf4-0ca55eefe7be" Jul 15 13:18:31.492981 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.492958 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-686cf7947c-8gr4l"] Jul 15 13:18:31.495725 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.495699 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.497961 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.497942 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 15 13:18:31.504895 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.504876 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-686cf7947c-8gr4l"] Jul 15 13:18:31.526656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.526632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rwr\" (UniqueName: \"kubernetes.io/projected/964b2652-7069-443f-b133-764f23eddbac-kube-api-access-82rwr\") pod \"authorino-686cf7947c-8gr4l\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.526743 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.526666 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6p5\" (UniqueName: \"kubernetes.io/projected/022c8462-fceb-4a5b-8bf4-0ca55eefe7be-kube-api-access-dh6p5\") pod \"authorino-569544f5cc-hwm7x\" (UID: \"022c8462-fceb-4a5b-8bf4-0ca55eefe7be\") " pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:31.526743 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.526696 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964b2652-7069-443f-b133-764f23eddbac-tls-cert\") pod \"authorino-686cf7947c-8gr4l\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.534537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.534512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6p5\" (UniqueName: \"kubernetes.io/projected/022c8462-fceb-4a5b-8bf4-0ca55eefe7be-kube-api-access-dh6p5\") pod \"authorino-569544f5cc-hwm7x\" (UID: \"022c8462-fceb-4a5b-8bf4-0ca55eefe7be\") " pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:31.608039 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.607975 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5w66" event={"ID":"0f96334a-483f-47f1-b2cc-ac131282c674","Type":"ContainerStarted","Data":"82555ccdf7489ec42b3be32ca55ab4b2d1e38d024303057a5bde9c73fb522ad7"} Jul 15 13:18:31.608039 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.608028 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:31.614010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.613991 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:31.627612 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.627592 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dh6p5\" (UniqueName: \"kubernetes.io/projected/022c8462-fceb-4a5b-8bf4-0ca55eefe7be-kube-api-access-dh6p5\") pod \"022c8462-fceb-4a5b-8bf4-0ca55eefe7be\" (UID: \"022c8462-fceb-4a5b-8bf4-0ca55eefe7be\") " Jul 15 13:18:31.627746 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.627730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rwr\" (UniqueName: \"kubernetes.io/projected/964b2652-7069-443f-b133-764f23eddbac-kube-api-access-82rwr\") pod \"authorino-686cf7947c-8gr4l\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.627784 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.627776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964b2652-7069-443f-b133-764f23eddbac-tls-cert\") pod \"authorino-686cf7947c-8gr4l\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.629877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.629845 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022c8462-fceb-4a5b-8bf4-0ca55eefe7be-kube-api-access-dh6p5" (OuterVolumeSpecName: "kube-api-access-dh6p5") pod "022c8462-fceb-4a5b-8bf4-0ca55eefe7be" (UID: "022c8462-fceb-4a5b-8bf4-0ca55eefe7be"). InnerVolumeSpecName "kube-api-access-dh6p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:18:31.630362 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.630342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964b2652-7069-443f-b133-764f23eddbac-tls-cert\") pod \"authorino-686cf7947c-8gr4l\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.635665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.635649 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rwr\" (UniqueName: \"kubernetes.io/projected/964b2652-7069-443f-b133-764f23eddbac-kube-api-access-82rwr\") pod \"authorino-686cf7947c-8gr4l\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.729237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.729210 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6p5\" (UniqueName: \"kubernetes.io/projected/022c8462-fceb-4a5b-8bf4-0ca55eefe7be-kube-api-access-dh6p5\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:18:31.804920 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.804895 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:18:31.939600 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:31.939571 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-686cf7947c-8gr4l"] Jul 15 13:18:31.943766 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:18:31.943740 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod964b2652_7069_443f_b133_764f23eddbac.slice/crio-4ac298e13b53018f270359e807aff328089c9a4f680a4c54050482cf627b9192 WatchSource:0}: Error finding container 4ac298e13b53018f270359e807aff328089c9a4f680a4c54050482cf627b9192: Status 404 returned error can't find the container with id 4ac298e13b53018f270359e807aff328089c9a4f680a4c54050482cf627b9192 Jul 15 13:18:32.620789 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.620756 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5w66" event={"ID":"0f96334a-483f-47f1-b2cc-ac131282c674","Type":"ContainerStarted","Data":"c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610"} Jul 15 13:18:32.621222 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.620808 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-s5w66" podUID="0f96334a-483f-47f1-b2cc-ac131282c674" containerName="authorino" containerID="cri-o://c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610" gracePeriod=30 Jul 15 13:18:32.622416 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.622393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686cf7947c-8gr4l" event={"ID":"964b2652-7069-443f-b133-764f23eddbac","Type":"ContainerStarted","Data":"1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93"} Jul 15 13:18:32.622569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.622424 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686cf7947c-8gr4l" event={"ID":"964b2652-7069-443f-b133-764f23eddbac","Type":"ContainerStarted","Data":"4ac298e13b53018f270359e807aff328089c9a4f680a4c54050482cf627b9192"} Jul 15 13:18:32.622569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.622522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-569544f5cc-hwm7x" Jul 15 13:18:32.637139 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.637098 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-s5w66" podStartSLOduration=1.298075233 podStartE2EDuration="1.637085172s" totalImagesPullingTime="339.009939ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:18:32.636455506 +0000 UTC m=+718.591910071" watchObservedRunningTime="2026-07-15 13:18:32.637085172 +0000 UTC m=+718.592539734" Jul 15 13:18:32.658807 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.658767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-686cf7947c-8gr4l" podStartSLOduration=1.319309158 podStartE2EDuration="1.658754107s" totalImagesPullingTime="339.444949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:18:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:18:32.657031544 +0000 UTC m=+718.612486106" watchObservedRunningTime="2026-07-15 13:18:32.658754107 +0000 UTC m=+718.614208670" Jul 15 13:18:32.687788 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.687762 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hkb7w"] Jul 15 13:18:32.687931 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.687912 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-hkb7w" podUID="48115549-8eaf-4212-bfff-407de9d8dbd6" containerName="authorino" containerID="cri-o://ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4" gracePeriod=30 Jul 15 13:18:32.710325 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.710298 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-569544f5cc-hwm7x"] Jul 15 13:18:32.717665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.717642 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-569544f5cc-hwm7x"] Jul 15 13:18:32.936786 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.936758 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:32.943054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:32.943034 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:33.041810 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.041780 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pk5ww\" (UniqueName: \"kubernetes.io/projected/0f96334a-483f-47f1-b2cc-ac131282c674-kube-api-access-pk5ww\") pod \"0f96334a-483f-47f1-b2cc-ac131282c674\" (UID: \"0f96334a-483f-47f1-b2cc-ac131282c674\") " Jul 15 13:18:33.041965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.041848 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts4t4\" (UniqueName: \"kubernetes.io/projected/48115549-8eaf-4212-bfff-407de9d8dbd6-kube-api-access-ts4t4\") pod \"48115549-8eaf-4212-bfff-407de9d8dbd6\" (UID: \"48115549-8eaf-4212-bfff-407de9d8dbd6\") " Jul 15 13:18:33.043873 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.043839 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f96334a-483f-47f1-b2cc-ac131282c674-kube-api-access-pk5ww" (OuterVolumeSpecName: "kube-api-access-pk5ww") pod "0f96334a-483f-47f1-b2cc-ac131282c674" (UID: "0f96334a-483f-47f1-b2cc-ac131282c674"). InnerVolumeSpecName "kube-api-access-pk5ww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:18:33.043985 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.043931 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48115549-8eaf-4212-bfff-407de9d8dbd6-kube-api-access-ts4t4" (OuterVolumeSpecName: "kube-api-access-ts4t4") pod "48115549-8eaf-4212-bfff-407de9d8dbd6" (UID: "48115549-8eaf-4212-bfff-407de9d8dbd6"). InnerVolumeSpecName "kube-api-access-ts4t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:18:33.143089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.143018 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4t4\" (UniqueName: \"kubernetes.io/projected/48115549-8eaf-4212-bfff-407de9d8dbd6-kube-api-access-ts4t4\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:18:33.143089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.143042 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pk5ww\" (UniqueName: \"kubernetes.io/projected/0f96334a-483f-47f1-b2cc-ac131282c674-kube-api-access-pk5ww\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:18:33.628418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.628382 2564 generic.go:356] "Generic (PLEG): container finished" podID="0f96334a-483f-47f1-b2cc-ac131282c674" containerID="c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610" exitCode=0 Jul 15 13:18:33.628418 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.628421 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-s5w66" Jul 15 13:18:33.628899 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.628465 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5w66" event={"ID":"0f96334a-483f-47f1-b2cc-ac131282c674","Type":"ContainerDied","Data":"c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610"} Jul 15 13:18:33.628899 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.628521 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-s5w66" event={"ID":"0f96334a-483f-47f1-b2cc-ac131282c674","Type":"ContainerDied","Data":"82555ccdf7489ec42b3be32ca55ab4b2d1e38d024303057a5bde9c73fb522ad7"} Jul 15 13:18:33.628899 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.628543 2564 scope.go:117] "RemoveContainer" containerID="c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610" Jul 15 13:18:33.629823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.629806 2564 generic.go:356] "Generic (PLEG): container finished" podID="48115549-8eaf-4212-bfff-407de9d8dbd6" containerID="ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4" exitCode=0 Jul 15 13:18:33.629901 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.629849 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-hkb7w" Jul 15 13:18:33.629901 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.629891 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hkb7w" event={"ID":"48115549-8eaf-4212-bfff-407de9d8dbd6","Type":"ContainerDied","Data":"ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4"} Jul 15 13:18:33.629985 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.629921 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-hkb7w" event={"ID":"48115549-8eaf-4212-bfff-407de9d8dbd6","Type":"ContainerDied","Data":"74c62890b409ec0cb5e3ca501950a555e101ba5903e677a01ec3b630c841cc20"} Jul 15 13:18:33.637212 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.637195 2564 scope.go:117] "RemoveContainer" containerID="c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610" Jul 15 13:18:33.637463 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:18:33.637441 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610\": container with ID starting with c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610 not found: ID does not exist" containerID="c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610" Jul 15 13:18:33.637546 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.637511 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610"} err="failed to get container status \"c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610\": rpc error: code = NotFound desc = could not find container \"c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610\": container with ID starting with c766fd12469194d3d14aefa0410939f1e8320226620c3411e2f4a94e36c97610 not found: ID does not exist" Jul 15 13:18:33.637546 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.637539 2564 scope.go:117] "RemoveContainer" containerID="ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4" Jul 15 13:18:33.645329 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.645314 2564 scope.go:117] "RemoveContainer" containerID="ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4" Jul 15 13:18:33.645610 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:18:33.645591 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4\": container with ID starting with ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4 not found: ID does not exist" containerID="ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4" Jul 15 13:18:33.645677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.645616 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4"} err="failed to get container status \"ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4\": rpc error: code = NotFound desc = could not find container \"ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4\": container with ID starting with ad1d154bdc589ac073bf54c3738d9012d2e1cbff7e74304d71782fe39883dbf4 not found: ID does not exist" Jul 15 13:18:33.654313 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.654286 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hkb7w"] Jul 15 13:18:33.660601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.660581 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-hkb7w"] Jul 15 13:18:33.671351 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.671329 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5w66"] Jul 15 13:18:33.675712 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:33.675694 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-s5w66"] Jul 15 13:18:34.656812 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:34.656784 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="022c8462-fceb-4a5b-8bf4-0ca55eefe7be" path="/var/lib/kubelet/pods/022c8462-fceb-4a5b-8bf4-0ca55eefe7be/volumes" Jul 15 13:18:34.657161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:34.657022 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f96334a-483f-47f1-b2cc-ac131282c674" path="/var/lib/kubelet/pods/0f96334a-483f-47f1-b2cc-ac131282c674/volumes" Jul 15 13:18:34.657310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:18:34.657298 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48115549-8eaf-4212-bfff-407de9d8dbd6" path="/var/lib/kubelet/pods/48115549-8eaf-4212-bfff-407de9d8dbd6/volumes" Jul 15 13:19:14.548615 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.548585 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8"] Jul 15 13:19:14.549097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.548963 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f96334a-483f-47f1-b2cc-ac131282c674" containerName="authorino" Jul 15 13:19:14.549097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.548982 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f96334a-483f-47f1-b2cc-ac131282c674" containerName="authorino" Jul 15 13:19:14.549097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.548996 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48115549-8eaf-4212-bfff-407de9d8dbd6" containerName="authorino" Jul 15 13:19:14.549097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.549001 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="48115549-8eaf-4212-bfff-407de9d8dbd6" containerName="authorino" Jul 15 13:19:14.549285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.549100 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="48115549-8eaf-4212-bfff-407de9d8dbd6" containerName="authorino" Jul 15 13:19:14.549285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.549115 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f96334a-483f-47f1-b2cc-ac131282c674" containerName="authorino" Jul 15 13:19:14.552382 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.552359 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.554822 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.554800 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 15 13:19:14.554929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.554831 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-k5mg5\"" Jul 15 13:19:14.563316 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.563294 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8"] Jul 15 13:19:14.608168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.608141 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cgs5\" (UniqueName: \"kubernetes.io/projected/5b80e131-71e9-41ed-81cf-d6530500f3bc-kube-api-access-4cgs5\") pod \"payload-pre-processing-74584bb9f8-jrrl8\" (UID: \"5b80e131-71e9-41ed-81cf-d6530500f3bc\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.608279 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.608193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80e131-71e9-41ed-81cf-d6530500f3bc-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-jrrl8\" (UID: \"5b80e131-71e9-41ed-81cf-d6530500f3bc\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.709257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.709226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80e131-71e9-41ed-81cf-d6530500f3bc-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-jrrl8\" (UID: \"5b80e131-71e9-41ed-81cf-d6530500f3bc\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.709399 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.709311 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cgs5\" (UniqueName: \"kubernetes.io/projected/5b80e131-71e9-41ed-81cf-d6530500f3bc-kube-api-access-4cgs5\") pod \"payload-pre-processing-74584bb9f8-jrrl8\" (UID: \"5b80e131-71e9-41ed-81cf-d6530500f3bc\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.709961 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.709940 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b80e131-71e9-41ed-81cf-d6530500f3bc-plugins-config-volume\") pod \"payload-pre-processing-74584bb9f8-jrrl8\" (UID: \"5b80e131-71e9-41ed-81cf-d6530500f3bc\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.718073 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.718049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cgs5\" (UniqueName: \"kubernetes.io/projected/5b80e131-71e9-41ed-81cf-d6530500f3bc-kube-api-access-4cgs5\") pod \"payload-pre-processing-74584bb9f8-jrrl8\" (UID: \"5b80e131-71e9-41ed-81cf-d6530500f3bc\") " pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.864378 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.864313 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:14.990152 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:14.990122 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8"] Jul 15 13:19:14.992535 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:19:14.992506 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b80e131_71e9_41ed_81cf_d6530500f3bc.slice/crio-7131d17a972e142b19ab1534271089222e81c1706b8e766e4a0ebeb0e8ace760 WatchSource:0}: Error finding container 7131d17a972e142b19ab1534271089222e81c1706b8e766e4a0ebeb0e8ace760: Status 404 returned error can't find the container with id 7131d17a972e142b19ab1534271089222e81c1706b8e766e4a0ebeb0e8ace760 Jul 15 13:19:15.820767 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:15.820729 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" event={"ID":"5b80e131-71e9-41ed-81cf-d6530500f3bc","Type":"ContainerStarted","Data":"7131d17a972e142b19ab1534271089222e81c1706b8e766e4a0ebeb0e8ace760"} Jul 15 13:19:16.826812 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:16.826778 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" event={"ID":"5b80e131-71e9-41ed-81cf-d6530500f3bc","Type":"ContainerStarted","Data":"752324fbc8f075217c628379c1bc1e213826095d3f56ae8afdfcdf7e1ef28e6f"} Jul 15 13:19:16.827224 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:16.826932 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:16.843003 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:16.842957 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" podStartSLOduration=1.343668963 podStartE2EDuration="2.842939453s" totalImagesPullingTime="1.49927049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:19:16.842016081 +0000 UTC m=+762.797470644" watchObservedRunningTime="2026-07-15 13:19:16.842939453 +0000 UTC m=+762.798394017" Jul 15 13:19:27.834242 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:27.834213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-74584bb9f8-jrrl8" Jul 15 13:19:35.627412 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.627374 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75"] Jul 15 13:19:35.630810 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.630795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.635274 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.635252 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 15 13:19:35.635274 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.635268 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 15 13:19:35.635492 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.635270 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6wk5z\"" Jul 15 13:19:35.635492 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.635252 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 15 13:19:35.663054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.663021 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75"] Jul 15 13:19:35.694028 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.693988 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.694195 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.694059 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.694195 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.694087 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v44d8\" (UniqueName: \"kubernetes.io/projected/83774881-a7a0-4cf9-834e-bd003a90ed29-kube-api-access-v44d8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.694313 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.694185 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83774881-a7a0-4cf9-834e-bd003a90ed29-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.694313 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.694258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.694407 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.694389 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.795249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795202 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.795445 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795427 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.795561 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v44d8\" (UniqueName: \"kubernetes.io/projected/83774881-a7a0-4cf9-834e-bd003a90ed29-kube-api-access-v44d8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.795647 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83774881-a7a0-4cf9-834e-bd003a90ed29-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.795696 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795686 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.795923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795896 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.796089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.795895 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.796337 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.796315 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.796546 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.796526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.797680 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.797660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/83774881-a7a0-4cf9-834e-bd003a90ed29-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.798027 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.798007 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/83774881-a7a0-4cf9-834e-bd003a90ed29-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.804034 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.804013 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v44d8\" (UniqueName: \"kubernetes.io/projected/83774881-a7a0-4cf9-834e-bd003a90ed29-kube-api-access-v44d8\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75\" (UID: \"83774881-a7a0-4cf9-834e-bd003a90ed29\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:35.941817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:35.941742 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:36.088857 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:36.088831 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75"] Jul 15 13:19:36.089429 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:19:36.089400 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83774881_a7a0_4cf9_834e_bd003a90ed29.slice/crio-54932adcfc2b574ba1ce5dd727db55fbdb52b8d6daada269f3563621f44a6297 WatchSource:0}: Error finding container 54932adcfc2b574ba1ce5dd727db55fbdb52b8d6daada269f3563621f44a6297: Status 404 returned error can't find the container with id 54932adcfc2b574ba1ce5dd727db55fbdb52b8d6daada269f3563621f44a6297 Jul 15 13:19:36.924050 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:36.924003 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" event={"ID":"83774881-a7a0-4cf9-834e-bd003a90ed29","Type":"ContainerStarted","Data":"54932adcfc2b574ba1ce5dd727db55fbdb52b8d6daada269f3563621f44a6297"} Jul 15 13:19:37.931294 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:37.931256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" event={"ID":"83774881-a7a0-4cf9-834e-bd003a90ed29","Type":"ContainerStarted","Data":"c2c3acea701eaf0ea0c1d696de6a30de95d8947b67e9622269b8a8cda86cf994"} Jul 15 13:19:37.954009 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:37.953963 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" podStartSLOduration=1.711218685 podStartE2EDuration="2.953947152s" totalImagesPullingTime="1.242728467s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:19:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:19:37.953628044 +0000 UTC m=+783.909082606" watchObservedRunningTime="2026-07-15 13:19:37.953947152 +0000 UTC m=+783.909401715" Jul 15 13:19:45.942243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:45.942197 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:45.942243 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:45.942257 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:45.955131 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:45.955101 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:45.983936 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:45.983907 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75" Jul 15 13:19:52.113242 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.113208 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx"] Jul 15 13:19:52.117865 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.117849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.120586 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.120563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 15 13:19:52.133811 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.133787 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx"] Jul 15 13:19:52.246421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.246388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.246421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.246422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.246666 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.246451 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba6581a-79aa-4320-ab58-9df4b8a113d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.246666 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.246576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7mk\" (UniqueName: \"kubernetes.io/projected/fba6581a-79aa-4320-ab58-9df4b8a113d3-kube-api-access-4l7mk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.246666 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.246632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.246666 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.246654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347499 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347780 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba6581a-79aa-4320-ab58-9df4b8a113d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347780 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347595 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7mk\" (UniqueName: \"kubernetes.io/projected/fba6581a-79aa-4320-ab58-9df4b8a113d3-kube-api-access-4l7mk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347780 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347780 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.347982 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.347945 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.348060 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.348029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.348169 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.348067 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.349896 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.349867 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fba6581a-79aa-4320-ab58-9df4b8a113d3-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.350132 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.350112 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fba6581a-79aa-4320-ab58-9df4b8a113d3-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.359999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.359979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7mk\" (UniqueName: \"kubernetes.io/projected/fba6581a-79aa-4320-ab58-9df4b8a113d3-kube-api-access-4l7mk\") pod \"e2e-trlp-test-simulated-kserve-598466548d-x98xx\" (UID: \"fba6581a-79aa-4320-ab58-9df4b8a113d3\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.427301 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.427233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:19:52.584658 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:52.584496 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx"] Jul 15 13:19:52.587330 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:19:52.587299 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba6581a_79aa_4320_ab58_9df4b8a113d3.slice/crio-27151618892a9ecb8e6be2f62ea21d2ff3fdca8714a3ca7605f90323eddf99ba WatchSource:0}: Error finding container 27151618892a9ecb8e6be2f62ea21d2ff3fdca8714a3ca7605f90323eddf99ba: Status 404 returned error can't find the container with id 27151618892a9ecb8e6be2f62ea21d2ff3fdca8714a3ca7605f90323eddf99ba Jul 15 13:19:53.006719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:53.006684 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" event={"ID":"fba6581a-79aa-4320-ab58-9df4b8a113d3","Type":"ContainerStarted","Data":"7ee3991e4bda1fe11ba649b371e5ae94fb4a9f5e9bc9b69d44265ca696352e3d"} Jul 15 13:19:53.006880 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:53.006741 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" event={"ID":"fba6581a-79aa-4320-ab58-9df4b8a113d3","Type":"ContainerStarted","Data":"27151618892a9ecb8e6be2f62ea21d2ff3fdca8714a3ca7605f90323eddf99ba"} Jul 15 13:19:53.034057 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:19:53.034006 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" podStartSLOduration=0.7969008 podStartE2EDuration="1.033991259s" totalImagesPullingTime="237.090459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:19:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:19:53.032225957 +0000 UTC m=+798.987680569" watchObservedRunningTime="2026-07-15 13:19:53.033991259 +0000 UTC m=+798.989445842" Jul 15 13:20:02.427989 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:20:02.427912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:20:02.427989 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:20:02.427967 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:20:02.440140 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:20:02.440119 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:20:03.067220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:20:03.067193 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-x98xx" Jul 15 13:21:17.606812 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.606778 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-856d6647d4-gxvs2"] Jul 15 13:21:17.610428 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.610405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.617803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.617779 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-856d6647d4-gxvs2"] Jul 15 13:21:17.716610 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.716574 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8816b4bc-ad61-470d-acc4-1218690a4730-tls-cert\") pod \"authorino-856d6647d4-gxvs2\" (UID: \"8816b4bc-ad61-470d-acc4-1218690a4730\") " pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.716776 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.716624 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb7lz\" (UniqueName: \"kubernetes.io/projected/8816b4bc-ad61-470d-acc4-1218690a4730-kube-api-access-fb7lz\") pod \"authorino-856d6647d4-gxvs2\" (UID: \"8816b4bc-ad61-470d-acc4-1218690a4730\") " pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.818186 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.818153 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8816b4bc-ad61-470d-acc4-1218690a4730-tls-cert\") pod \"authorino-856d6647d4-gxvs2\" (UID: \"8816b4bc-ad61-470d-acc4-1218690a4730\") " pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.818347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.818194 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb7lz\" (UniqueName: \"kubernetes.io/projected/8816b4bc-ad61-470d-acc4-1218690a4730-kube-api-access-fb7lz\") pod \"authorino-856d6647d4-gxvs2\" (UID: \"8816b4bc-ad61-470d-acc4-1218690a4730\") " pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.820753 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.820731 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/8816b4bc-ad61-470d-acc4-1218690a4730-tls-cert\") pod \"authorino-856d6647d4-gxvs2\" (UID: \"8816b4bc-ad61-470d-acc4-1218690a4730\") " pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.826537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.826518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb7lz\" (UniqueName: \"kubernetes.io/projected/8816b4bc-ad61-470d-acc4-1218690a4730-kube-api-access-fb7lz\") pod \"authorino-856d6647d4-gxvs2\" (UID: \"8816b4bc-ad61-470d-acc4-1218690a4730\") " pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:17.920874 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:17.920791 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-856d6647d4-gxvs2" Jul 15 13:21:18.056805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:18.056777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-856d6647d4-gxvs2"] Jul 15 13:21:18.058708 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:21:18.058676 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8816b4bc_ad61_470d_acc4_1218690a4730.slice/crio-f355947cd08ae33a22b1e4f3e0aae10a80e88e8c9f3a31a4933f3ce9ab99418d WatchSource:0}: Error finding container f355947cd08ae33a22b1e4f3e0aae10a80e88e8c9f3a31a4933f3ce9ab99418d: Status 404 returned error can't find the container with id f355947cd08ae33a22b1e4f3e0aae10a80e88e8c9f3a31a4933f3ce9ab99418d Jul 15 13:21:18.417850 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:18.417805 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856d6647d4-gxvs2" event={"ID":"8816b4bc-ad61-470d-acc4-1218690a4730","Type":"ContainerStarted","Data":"f355947cd08ae33a22b1e4f3e0aae10a80e88e8c9f3a31a4933f3ce9ab99418d"} Jul 15 13:21:19.424907 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.424874 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-856d6647d4-gxvs2" event={"ID":"8816b4bc-ad61-470d-acc4-1218690a4730","Type":"ContainerStarted","Data":"fed8c9cc6e771b3e91d58259cf81ab5e68c88cb8633299be5b8c681ab34d8a3e"} Jul 15 13:21:19.455526 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.455448 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-856d6647d4-gxvs2" podStartSLOduration=1.889475703 podStartE2EDuration="2.455426272s" totalImagesPullingTime="565.950569ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:21:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:21:19.45106401 +0000 UTC m=+885.406518612" watchObservedRunningTime="2026-07-15 13:21:19.455426272 +0000 UTC m=+885.410880837" Jul 15 13:21:19.485847 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.485812 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686cf7947c-8gr4l"] Jul 15 13:21:19.486170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.486141 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-686cf7947c-8gr4l" podUID="964b2652-7069-443f-b133-764f23eddbac" containerName="authorino" containerID="cri-o://1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93" gracePeriod=30 Jul 15 13:21:19.746131 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.746093 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:21:19.838348 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.838310 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82rwr\" (UniqueName: \"kubernetes.io/projected/964b2652-7069-443f-b133-764f23eddbac-kube-api-access-82rwr\") pod \"964b2652-7069-443f-b133-764f23eddbac\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " Jul 15 13:21:19.838551 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.838436 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964b2652-7069-443f-b133-764f23eddbac-tls-cert\") pod \"964b2652-7069-443f-b133-764f23eddbac\" (UID: \"964b2652-7069-443f-b133-764f23eddbac\") " Jul 15 13:21:19.840451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.840419 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b2652-7069-443f-b133-764f23eddbac-kube-api-access-82rwr" (OuterVolumeSpecName: "kube-api-access-82rwr") pod "964b2652-7069-443f-b133-764f23eddbac" (UID: "964b2652-7069-443f-b133-764f23eddbac"). InnerVolumeSpecName "kube-api-access-82rwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:21:19.849655 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.849622 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/964b2652-7069-443f-b133-764f23eddbac-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "964b2652-7069-443f-b133-764f23eddbac" (UID: "964b2652-7069-443f-b133-764f23eddbac"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:21:19.939548 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.939519 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rwr\" (UniqueName: \"kubernetes.io/projected/964b2652-7069-443f-b133-764f23eddbac-kube-api-access-82rwr\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:21:19.939548 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:19.939545 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/964b2652-7069-443f-b133-764f23eddbac-tls-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:21:20.431018 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.430978 2564 generic.go:356] "Generic (PLEG): container finished" podID="964b2652-7069-443f-b133-764f23eddbac" containerID="1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93" exitCode=0 Jul 15 13:21:20.431513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.431025 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-686cf7947c-8gr4l" Jul 15 13:21:20.431513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.431058 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686cf7947c-8gr4l" event={"ID":"964b2652-7069-443f-b133-764f23eddbac","Type":"ContainerDied","Data":"1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93"} Jul 15 13:21:20.431513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.431096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-686cf7947c-8gr4l" event={"ID":"964b2652-7069-443f-b133-764f23eddbac","Type":"ContainerDied","Data":"4ac298e13b53018f270359e807aff328089c9a4f680a4c54050482cf627b9192"} Jul 15 13:21:20.431513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.431115 2564 scope.go:117] "RemoveContainer" containerID="1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93" Jul 15 13:21:20.445376 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.445349 2564 scope.go:117] "RemoveContainer" containerID="1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93" Jul 15 13:21:20.445718 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:21:20.445695 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93\": container with ID starting with 1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93 not found: ID does not exist" containerID="1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93" Jul 15 13:21:20.445807 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.445728 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93"} err="failed to get container status \"1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93\": rpc error: code = NotFound desc = could not find container \"1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93\": container with ID starting with 1b6d760ce104d2e72822bbbe99b14e8e75398bbe4e0d7eb3ee1b5602df596e93 not found: ID does not exist" Jul 15 13:21:20.454180 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.454157 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-686cf7947c-8gr4l"] Jul 15 13:21:20.459373 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.459348 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-686cf7947c-8gr4l"] Jul 15 13:21:20.658962 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:20.658930 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="964b2652-7069-443f-b133-764f23eddbac" path="/var/lib/kubelet/pods/964b2652-7069-443f-b133-764f23eddbac/volumes" Jul 15 13:21:34.556896 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:34.556873 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:21:34.557991 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:21:34.557973 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:26:34.589130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:26:34.589096 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:26:34.590998 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:26:34.590973 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:30:00.148595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.148559 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4"] Jul 15 13:30:00.149099 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.148968 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="964b2652-7069-443f-b133-764f23eddbac" containerName="authorino" Jul 15 13:30:00.149099 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.148979 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="964b2652-7069-443f-b133-764f23eddbac" containerName="authorino" Jul 15 13:30:00.149099 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.149064 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="964b2652-7069-443f-b133-764f23eddbac" containerName="authorino" Jul 15 13:30:00.152247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.152226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" Jul 15 13:30:00.155714 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.155690 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:30:00.160660 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.160629 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4"] Jul 15 13:30:00.163247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.163226 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" Jul 15 13:30:00.295899 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.295875 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4"] Jul 15 13:30:00.297677 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:30:00.297649 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ffe8c5_6286_4662_b8c4_11a1aee1a704.slice/crio-2bdc52b6fa3e7816e12506fe61e76c3bba8a08a70a9f6f4187e5c990530da268 WatchSource:0}: Error finding container 2bdc52b6fa3e7816e12506fe61e76c3bba8a08a70a9f6f4187e5c990530da268: Status 404 returned error can't find the container with id 2bdc52b6fa3e7816e12506fe61e76c3bba8a08a70a9f6f4187e5c990530da268 Jul 15 13:30:00.299609 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.299591 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:30:00.967914 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:00.967876 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" event={"ID":"72ffe8c5-6286-4662-b8c4-11a1aee1a704","Type":"ContainerStarted","Data":"2bdc52b6fa3e7816e12506fe61e76c3bba8a08a70a9f6f4187e5c990530da268"} Jul 15 13:30:02.980037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:02.980004 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" event={"ID":"72ffe8c5-6286-4662-b8c4-11a1aee1a704","Type":"ContainerStarted","Data":"fc8297354e07a2ee1e48bd93ead391dd3ead52333d54ce4a965c4f64ebce980f"} Jul 15 13:30:02.996427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:02.996380 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" podStartSLOduration=0.781048453 podStartE2EDuration="2.996363806s" totalImagesPullingTime="2.215315353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:30:02.995191018 +0000 UTC m=+1408.950645582" watchObservedRunningTime="2026-07-15 13:30:02.996363806 +0000 UTC m=+1408.951818390" Jul 15 13:30:03.995811 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:03.995769 2564 generic.go:356] "Generic (PLEG): container finished" podID="72ffe8c5-6286-4662-b8c4-11a1aee1a704" containerID="fc8297354e07a2ee1e48bd93ead391dd3ead52333d54ce4a965c4f64ebce980f" exitCode=0 Jul 15 13:30:03.996277 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:03.995903 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" event={"ID":"72ffe8c5-6286-4662-b8c4-11a1aee1a704","Type":"ContainerDied","Data":"fc8297354e07a2ee1e48bd93ead391dd3ead52333d54ce4a965c4f64ebce980f"} Jul 15 13:30:05.131776 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:05.131751 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" Jul 15 13:30:06.006541 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:06.006510 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" Jul 15 13:30:06.006717 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:06.006536 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735370-xx8c4" event={"ID":"72ffe8c5-6286-4662-b8c4-11a1aee1a704","Type":"ContainerDied","Data":"2bdc52b6fa3e7816e12506fe61e76c3bba8a08a70a9f6f4187e5c990530da268"} Jul 15 13:30:06.006717 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:30:06.006584 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bdc52b6fa3e7816e12506fe61e76c3bba8a08a70a9f6f4187e5c990530da268" Jul 15 13:31:34.621246 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:31:34.621217 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:31:34.623600 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:31:34.623580 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:34:26.048912 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.048879 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826"] Jul 15 13:34:26.049309 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.049290 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ffe8c5-6286-4662-b8c4-11a1aee1a704" containerName="cleanup" Jul 15 13:34:26.049309 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.049301 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ffe8c5-6286-4662-b8c4-11a1aee1a704" containerName="cleanup" Jul 15 13:34:26.049391 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.049385 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ffe8c5-6286-4662-b8c4-11a1aee1a704" containerName="cleanup" Jul 15 13:34:26.051430 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.051414 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:26.053710 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.053684 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:34:26.053710 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.053705 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:34:26.060881 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.060859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826"] Jul 15 13:34:26.107087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.107061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb3eae66-bk826\" (UID: \"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:26.207863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.207835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb3eae66-bk826\" (UID: \"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:26.208405 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.208388 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb3eae66-bk826\" (UID: \"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:26.363042 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.362977 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:26.695808 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:26.695779 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826"] Jul 15 13:34:26.697229 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:34:26.697201 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc26cb4df_2337_4c3b_9cd6_2c82b82d56f3.slice/crio-b8c646a4321c63786c82737cd17606373533907dae6a53b5a7c07b5d87fbfd87 WatchSource:0}: Error finding container b8c646a4321c63786c82737cd17606373533907dae6a53b5a7c07b5d87fbfd87: Status 404 returned error can't find the container with id b8c646a4321c63786c82737cd17606373533907dae6a53b5a7c07b5d87fbfd87 Jul 15 13:34:27.309008 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:27.308965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" event={"ID":"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3","Type":"ContainerStarted","Data":"868b471113c71c977484102143f94dd53691008725d1f0842b7b2d87949c87c0"} Jul 15 13:34:27.309342 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:27.309026 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" event={"ID":"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3","Type":"ContainerStarted","Data":"b8c646a4321c63786c82737cd17606373533907dae6a53b5a7c07b5d87fbfd87"} Jul 15 13:34:27.326796 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:27.326741 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" podStartSLOduration=1.326723092 podStartE2EDuration="1.326723092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:34:27.325589279 +0000 UTC m=+1673.281043842" watchObservedRunningTime="2026-07-15 13:34:27.326723092 +0000 UTC m=+1673.282177657" Jul 15 13:34:28.315263 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:28.315233 2564 generic.go:356] "Generic (PLEG): container finished" podID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerID="868b471113c71c977484102143f94dd53691008725d1f0842b7b2d87949c87c0" exitCode=7 Jul 15 13:34:28.315651 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:28.315284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" event={"ID":"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3","Type":"ContainerDied","Data":"868b471113c71c977484102143f94dd53691008725d1f0842b7b2d87949c87c0"} Jul 15 13:34:28.315651 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:28.315483 2564 scope.go:117] "RemoveContainer" containerID="868b471113c71c977484102143f94dd53691008725d1f0842b7b2d87949c87c0" Jul 15 13:34:29.321339 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:29.321311 2564 generic.go:356] "Generic (PLEG): container finished" podID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerID="ac5518ee06572e83de43828d0aebfe2d926369a891a6e29b9a1d1fcb4c6f0e90" exitCode=7 Jul 15 13:34:29.321697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:29.321387 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" event={"ID":"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3","Type":"ContainerDied","Data":"ac5518ee06572e83de43828d0aebfe2d926369a891a6e29b9a1d1fcb4c6f0e90"} Jul 15 13:34:29.321697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:29.321454 2564 scope.go:117] "RemoveContainer" containerID="868b471113c71c977484102143f94dd53691008725d1f0842b7b2d87949c87c0" Jul 15 13:34:29.321697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:29.321615 2564 scope.go:117] "RemoveContainer" containerID="ac5518ee06572e83de43828d0aebfe2d926369a891a6e29b9a1d1fcb4c6f0e90" Jul 15 13:34:29.321842 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:34:29.321825 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fb3eae66-bk826_odh-ai-gateway-infra(c26cb4df-2337-4c3b-9cd6-2c82b82d56f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" Jul 15 13:34:30.327919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:30.327894 2564 scope.go:117] "RemoveContainer" containerID="ac5518ee06572e83de43828d0aebfe2d926369a891a6e29b9a1d1fcb4c6f0e90" Jul 15 13:34:30.328264 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:34:30.328062 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-fb3eae66-bk826_odh-ai-gateway-infra(c26cb4df-2337-4c3b-9cd6-2c82b82d56f3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" Jul 15 13:34:44.655031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:44.655005 2564 scope.go:117] "RemoveContainer" containerID="ac5518ee06572e83de43828d0aebfe2d926369a891a6e29b9a1d1fcb4c6f0e90" Jul 15 13:34:45.408811 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:45.408784 2564 generic.go:356] "Generic (PLEG): container finished" podID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerID="b1199bb48640ce2d0feec28f139d1c6b934b606087edf936ea4263739a04e775" exitCode=0 Jul 15 13:34:45.408948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:45.408862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" event={"ID":"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3","Type":"ContainerDied","Data":"b1199bb48640ce2d0feec28f139d1c6b934b606087edf936ea4263739a04e775"} Jul 15 13:34:45.408948 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:45.408936 2564 scope.go:117] "RemoveContainer" containerID="ac5518ee06572e83de43828d0aebfe2d926369a891a6e29b9a1d1fcb4c6f0e90" Jul 15 13:34:46.443023 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:46.442989 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826"] Jul 15 13:34:46.550353 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:46.550329 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:46.688276 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:46.688204 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3-maas-api-service-ca\") pod \"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3\" (UID: \"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3\") " Jul 15 13:34:46.688569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:46.688546 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" (UID: "c26cb4df-2337-4c3b-9cd6-2c82b82d56f3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:34:46.789552 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:46.789516 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:34:47.421851 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:47.421814 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" event={"ID":"c26cb4df-2337-4c3b-9cd6-2c82b82d56f3","Type":"ContainerDied","Data":"b8c646a4321c63786c82737cd17606373533907dae6a53b5a7c07b5d87fbfd87"} Jul 15 13:34:47.422030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:47.421862 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8c646a4321c63786c82737cd17606373533907dae6a53b5a7c07b5d87fbfd87" Jul 15 13:34:47.422030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:47.421822 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826" Jul 15 13:34:47.470751 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:47.470721 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826"] Jul 15 13:34:47.478448 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:47.478427 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-bk826"] Jul 15 13:34:48.656968 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:34:48.656941 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" path="/var/lib/kubelet/pods/c26cb4df-2337-4c3b-9cd6-2c82b82d56f3/volumes" Jul 15 13:35:26.103994 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.103962 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw"] Jul 15 13:35:26.104487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104367 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104378 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104397 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104402 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104413 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104418 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104496 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104507 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.104708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.104515 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c26cb4df-2337-4c3b-9cd6-2c82b82d56f3" containerName="revoke-keys" Jul 15 13:35:26.106611 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.106590 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:26.109318 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.109294 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:35:26.109419 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.109330 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:35:26.116260 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.116230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw"] Jul 15 13:35:26.221140 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.221113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw\" (UID: \"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:26.322235 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.322212 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw\" (UID: \"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:26.322828 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.322810 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw\" (UID: \"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:26.418031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.417979 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:26.551295 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.551269 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw"] Jul 15 13:35:26.553136 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:35:26.553115 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc5f0da_44e8_4ba2_adb9_e9dbaee4fcea.slice/crio-7c09a3c2719f05c1c9a2227b28e83d635947b9b864aa1a59ce9c9d31fc7ebd0a WatchSource:0}: Error finding container 7c09a3c2719f05c1c9a2227b28e83d635947b9b864aa1a59ce9c9d31fc7ebd0a: Status 404 returned error can't find the container with id 7c09a3c2719f05c1c9a2227b28e83d635947b9b864aa1a59ce9c9d31fc7ebd0a Jul 15 13:35:26.614316 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:26.614286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" event={"ID":"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea","Type":"ContainerStarted","Data":"7c09a3c2719f05c1c9a2227b28e83d635947b9b864aa1a59ce9c9d31fc7ebd0a"} Jul 15 13:35:27.620929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:27.620901 2564 generic.go:356] "Generic (PLEG): container finished" podID="ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea" containerID="723aef81e44b795bfc4eb2c906f94389d653385d98d113d9eb1c4d658e2cafb9" exitCode=0 Jul 15 13:35:27.621302 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:27.620967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" event={"ID":"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea","Type":"ContainerDied","Data":"723aef81e44b795bfc4eb2c906f94389d653385d98d113d9eb1c4d658e2cafb9"} Jul 15 13:35:28.768713 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:28.768690 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:28.843769 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:28.843741 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea-maas-api-service-ca\") pod \"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea\" (UID: \"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea\") " Jul 15 13:35:28.844075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:28.844053 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea" (UID: "ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:35:28.944523 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:28.944452 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:35:29.632388 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:29.632357 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" Jul 15 13:35:29.632388 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:29.632370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw" event={"ID":"ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea","Type":"ContainerDied","Data":"7c09a3c2719f05c1c9a2227b28e83d635947b9b864aa1a59ce9c9d31fc7ebd0a"} Jul 15 13:35:29.632607 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:29.632416 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c09a3c2719f05c1c9a2227b28e83d635947b9b864aa1a59ce9c9d31fc7ebd0a" Jul 15 13:35:52.051271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.051226 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq"] Jul 15 13:35:52.051884 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.051858 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea" containerName="revoke-keys" Jul 15 13:35:52.051884 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.051883 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea" containerName="revoke-keys" Jul 15 13:35:52.052068 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.051999 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea" containerName="revoke-keys" Jul 15 13:35:52.057697 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.057675 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.066730 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.066388 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:35:52.066959 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.066937 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-faa562cb-gw-openshift-default-dockercfg-mns87\"" Jul 15 13:35:52.121619 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.121592 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq"] Jul 15 13:35:52.152256 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152228 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-data\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-podinfo\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152333 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-certs\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152519 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152401 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njfhh\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-kube-api-access-njfhh\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152519 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-envoy\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152519 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152491 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-socket\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152688 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istiod-ca-cert\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152688 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152607 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-token\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.152688 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.152668 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-credential-socket\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254068 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254046 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-podinfo\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254078 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-certs\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254109 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njfhh\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-kube-api-access-njfhh\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254128 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-envoy\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254177 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254148 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-socket\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254341 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istiod-ca-cert\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254341 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254324 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-token\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254439 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-credential-socket\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254439 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-data\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254517 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-certs\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254567 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-socket\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254742 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-data\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.254862 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254764 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-credential-socket\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.255014 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.254994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istiod-ca-cert\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.256376 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.256357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-envoy\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.256544 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.256528 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-podinfo\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.272039 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.272018 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-token\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.274230 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.274213 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njfhh\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-kube-api-access-njfhh\") pod \"e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.370540 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.370462 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:52.512199 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.512176 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq"] Jul 15 13:35:52.514003 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:35:52.513975 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c414ea1_b32e_4342_bb7b_a1f28300b10c.slice/crio-6d9c22a7f01cc57dff7e6cd0d00a1b4d7641fea1d3014fe4bf7c4e78979f7a4a WatchSource:0}: Error finding container 6d9c22a7f01cc57dff7e6cd0d00a1b4d7641fea1d3014fe4bf7c4e78979f7a4a: Status 404 returned error can't find the container with id 6d9c22a7f01cc57dff7e6cd0d00a1b4d7641fea1d3014fe4bf7c4e78979f7a4a Jul 15 13:35:52.515925 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.515909 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:35:52.745211 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:52.745136 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" event={"ID":"0c414ea1-b32e-4342-bb7b-a1f28300b10c","Type":"ContainerStarted","Data":"6d9c22a7f01cc57dff7e6cd0d00a1b4d7641fea1d3014fe4bf7c4e78979f7a4a"} Jul 15 13:35:55.168308 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:55.168269 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:35:55.168703 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:55.168356 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:35:55.168703 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:55.168406 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:35:55.763211 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:55.763176 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" event={"ID":"0c414ea1-b32e-4342-bb7b-a1f28300b10c","Type":"ContainerStarted","Data":"3cd061ad43854985261e36ef9f54927d868a03c6341b0e375b4024c2a5380b8d"} Jul 15 13:35:55.785216 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:55.785172 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podStartSLOduration=1.133233316 podStartE2EDuration="3.785158486s" totalImagesPullingTime="2.65192517s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:35:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:35:55.78315858 +0000 UTC m=+1761.738613179" watchObservedRunningTime="2026-07-15 13:35:55.785158486 +0000 UTC m=+1761.740613048" Jul 15 13:35:56.371282 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:56.371248 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:56.375930 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:56.375905 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:56.767812 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:56.767780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:56.768772 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:56.768752 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:35:57.798038 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.798003 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28"] Jul 15 13:35:57.801590 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.801573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:35:57.803915 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.803894 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:35:57.804050 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.803990 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:35:57.806615 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.806589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90770255-ef0c-469a-9081-e4a2b9a278cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28\" (UID: \"90770255-ef0c-469a-9081-e4a2b9a278cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:35:57.809609 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.809584 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28"] Jul 15 13:35:57.907281 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.907256 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90770255-ef0c-469a-9081-e4a2b9a278cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28\" (UID: \"90770255-ef0c-469a-9081-e4a2b9a278cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:35:57.907863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:57.907848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90770255-ef0c-469a-9081-e4a2b9a278cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28\" (UID: \"90770255-ef0c-469a-9081-e4a2b9a278cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:35:58.112677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:58.112601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:35:58.242429 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:58.242402 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28"] Jul 15 13:35:58.243994 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:35:58.243963 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90770255_ef0c_469a_9081_e4a2b9a278cc.slice/crio-64f51d42e29ed45970c73f0d5d4172cbb93c7087c4f75acee160d7863a996a02 WatchSource:0}: Error finding container 64f51d42e29ed45970c73f0d5d4172cbb93c7087c4f75acee160d7863a996a02: Status 404 returned error can't find the container with id 64f51d42e29ed45970c73f0d5d4172cbb93c7087c4f75acee160d7863a996a02 Jul 15 13:35:58.778063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:58.777977 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" event={"ID":"90770255-ef0c-469a-9081-e4a2b9a278cc","Type":"ContainerStarted","Data":"2452607f7c61a125ece3e94a4f07f76dc797aa5ce5978153bf9d596a8fc31d1b"} Jul 15 13:35:58.778063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:58.778024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" event={"ID":"90770255-ef0c-469a-9081-e4a2b9a278cc","Type":"ContainerStarted","Data":"64f51d42e29ed45970c73f0d5d4172cbb93c7087c4f75acee160d7863a996a02"} Jul 15 13:35:59.783935 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:59.783905 2564 generic.go:356] "Generic (PLEG): container finished" podID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerID="2452607f7c61a125ece3e94a4f07f76dc797aa5ce5978153bf9d596a8fc31d1b" exitCode=7 Jul 15 13:35:59.784283 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:59.783983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" event={"ID":"90770255-ef0c-469a-9081-e4a2b9a278cc","Type":"ContainerDied","Data":"2452607f7c61a125ece3e94a4f07f76dc797aa5ce5978153bf9d596a8fc31d1b"} Jul 15 13:35:59.784283 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:35:59.784162 2564 scope.go:117] "RemoveContainer" containerID="2452607f7c61a125ece3e94a4f07f76dc797aa5ce5978153bf9d596a8fc31d1b" Jul 15 13:36:00.791118 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:00.791086 2564 generic.go:356] "Generic (PLEG): container finished" podID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerID="cd2fa7cc29a4738034a2a44e8e358b137071204a23c01a5401aa4de9fb1487f1" exitCode=7 Jul 15 13:36:00.791566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:00.791168 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" event={"ID":"90770255-ef0c-469a-9081-e4a2b9a278cc","Type":"ContainerDied","Data":"cd2fa7cc29a4738034a2a44e8e358b137071204a23c01a5401aa4de9fb1487f1"} Jul 15 13:36:00.791566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:00.791237 2564 scope.go:117] "RemoveContainer" containerID="2452607f7c61a125ece3e94a4f07f76dc797aa5ce5978153bf9d596a8fc31d1b" Jul 15 13:36:00.791566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:00.791347 2564 scope.go:117] "RemoveContainer" containerID="cd2fa7cc29a4738034a2a44e8e358b137071204a23c01a5401aa4de9fb1487f1" Jul 15 13:36:00.791728 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:36:00.791631 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28_odh-ai-gateway-infra(90770255-ef0c-469a-9081-e4a2b9a278cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" Jul 15 13:36:01.797903 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:01.797878 2564 scope.go:117] "RemoveContainer" containerID="cd2fa7cc29a4738034a2a44e8e358b137071204a23c01a5401aa4de9fb1487f1" Jul 15 13:36:01.798264 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:36:01.798074 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28_odh-ai-gateway-infra(90770255-ef0c-469a-9081-e4a2b9a278cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" Jul 15 13:36:09.704512 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:36:09.704482 2564 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@0-23013-0.service\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c414ea1_b32e_4342_bb7b_a1f28300b10c.slice/crio-3cd061ad43854985261e36ef9f54927d868a03c6341b0e375b4024c2a5380b8d.scope\": RecentStats: unable to find data in memory cache]" Jul 15 13:36:09.846617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:09.846557 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerID="3cd061ad43854985261e36ef9f54927d868a03c6341b0e375b4024c2a5380b8d" exitCode=0 Jul 15 13:36:09.846617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:09.846602 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" event={"ID":"0c414ea1-b32e-4342-bb7b-a1f28300b10c","Type":"ContainerDied","Data":"3cd061ad43854985261e36ef9f54927d868a03c6341b0e375b4024c2a5380b8d"} Jul 15 13:36:09.846976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:09.846961 2564 scope.go:117] "RemoveContainer" containerID="3cd061ad43854985261e36ef9f54927d868a03c6341b0e375b4024c2a5380b8d" Jul 15 13:36:09.848075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:09.848046 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:36:09.848156 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:09.848114 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:36:09.848156 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:09.848145 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:36:10.854574 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:10.854544 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" event={"ID":"0c414ea1-b32e-4342-bb7b-a1f28300b10c","Type":"ContainerStarted","Data":"359a467f5489819d95cde59899ffe3ca7b2e7d0b96169d53f8892db2c59655f7"} Jul 15 13:36:11.370652 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:11.370614 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:36:11.371748 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:11.371723 2564 patch_prober.go:28] interesting pod/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:36:11.371856 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:11.371783 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:36:11.767889 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:11.767806 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:36:12.371268 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:12.371236 2564 patch_prober.go:28] interesting pod/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:36:12.371679 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:12.371300 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:36:13.371736 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:13.371702 2564 patch_prober.go:28] interesting pod/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:36:13.372123 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:13.371758 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:36:13.652595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:13.652124 2564 scope.go:117] "RemoveContainer" containerID="cd2fa7cc29a4738034a2a44e8e358b137071204a23c01a5401aa4de9fb1487f1" Jul 15 13:36:14.371795 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:14.371758 2564 patch_prober.go:28] interesting pod/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Jul 15 13:36:14.372156 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:14.371814 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Jul 15 13:36:14.877657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:14.877624 2564 generic.go:356] "Generic (PLEG): container finished" podID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerID="df78d5b4f2d2ae3cab4ac12b8eabc0edf24946e14a92011039b07d1c96c638e7" exitCode=0 Jul 15 13:36:14.877823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:14.877699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" event={"ID":"90770255-ef0c-469a-9081-e4a2b9a278cc","Type":"ContainerDied","Data":"df78d5b4f2d2ae3cab4ac12b8eabc0edf24946e14a92011039b07d1c96c638e7"} Jul 15 13:36:14.877823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:14.877758 2564 scope.go:117] "RemoveContainer" containerID="cd2fa7cc29a4738034a2a44e8e358b137071204a23c01a5401aa4de9fb1487f1" Jul 15 13:36:15.374708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:15.374680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:36:15.375708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:15.375683 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:36:15.907551 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:15.907519 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28"] Jul 15 13:36:16.017651 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.017625 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:36:16.071357 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.071326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90770255-ef0c-469a-9081-e4a2b9a278cc-maas-api-service-ca\") pod \"90770255-ef0c-469a-9081-e4a2b9a278cc\" (UID: \"90770255-ef0c-469a-9081-e4a2b9a278cc\") " Jul 15 13:36:16.071678 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.071656 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90770255-ef0c-469a-9081-e4a2b9a278cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90770255-ef0c-469a-9081-e4a2b9a278cc" (UID: "90770255-ef0c-469a-9081-e4a2b9a278cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:36:16.172723 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.172670 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90770255-ef0c-469a-9081-e4a2b9a278cc-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:36:16.890345 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.890311 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" event={"ID":"90770255-ef0c-469a-9081-e4a2b9a278cc","Type":"ContainerDied","Data":"64f51d42e29ed45970c73f0d5d4172cbb93c7087c4f75acee160d7863a996a02"} Jul 15 13:36:16.890757 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.890379 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64f51d42e29ed45970c73f0d5d4172cbb93c7087c4f75acee160d7863a996a02" Jul 15 13:36:16.890757 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.890321 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28" Jul 15 13:36:16.922270 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.922209 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28"] Jul 15 13:36:16.929969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:16.929943 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-n4j28"] Jul 15 13:36:18.656734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:18.656703 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" path="/var/lib/kubelet/pods/90770255-ef0c-469a-9081-e4a2b9a278cc/volumes" Jul 15 13:36:34.659592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:34.659565 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:36:34.668968 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:34.668948 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:36:47.859413 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859377 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs"] Jul 15 13:36:47.859804 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859787 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:36:47.859804 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859800 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:36:47.859875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859810 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:36:47.859875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859816 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:36:47.859939 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859888 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:36:47.859939 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.859899 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:36:47.862111 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.862092 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:47.864569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.864547 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:36:47.864665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.864635 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:36:47.871217 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.871195 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs"] Jul 15 13:36:47.973803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:47.973779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a22eacc-89bb-427b-aba2-68e9bf1d8f9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs\" (UID: \"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:48.074423 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:48.074394 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a22eacc-89bb-427b-aba2-68e9bf1d8f9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs\" (UID: \"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:48.075060 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:48.075041 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a22eacc-89bb-427b-aba2-68e9bf1d8f9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs\" (UID: \"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:48.173825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:48.173754 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:48.514221 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:48.514192 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs"] Jul 15 13:36:48.515825 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:36:48.515795 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a22eacc_89bb_427b_aba2_68e9bf1d8f9c.slice/crio-d66ff6bb1fd24d90f1272e17e6428d9ad3a02c5714d6bc5169b56ee07569bebb WatchSource:0}: Error finding container d66ff6bb1fd24d90f1272e17e6428d9ad3a02c5714d6bc5169b56ee07569bebb: Status 404 returned error can't find the container with id d66ff6bb1fd24d90f1272e17e6428d9ad3a02c5714d6bc5169b56ee07569bebb Jul 15 13:36:49.054406 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:49.054369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" event={"ID":"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c","Type":"ContainerStarted","Data":"83ac2aaf924e8bb376b26476578792a517714692cd4ee712f85b388f45c4d602"} Jul 15 13:36:49.054822 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:49.054422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" event={"ID":"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c","Type":"ContainerStarted","Data":"d66ff6bb1fd24d90f1272e17e6428d9ad3a02c5714d6bc5169b56ee07569bebb"} Jul 15 13:36:49.070706 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:49.070666 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" podStartSLOduration=2.070651366 podStartE2EDuration="2.070651366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:36:49.068957036 +0000 UTC m=+1815.024411611" watchObservedRunningTime="2026-07-15 13:36:49.070651366 +0000 UTC m=+1815.026105928" Jul 15 13:36:50.061282 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:50.061250 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a22eacc-89bb-427b-aba2-68e9bf1d8f9c" containerID="83ac2aaf924e8bb376b26476578792a517714692cd4ee712f85b388f45c4d602" exitCode=0 Jul 15 13:36:50.061733 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:50.061333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" event={"ID":"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c","Type":"ContainerDied","Data":"83ac2aaf924e8bb376b26476578792a517714692cd4ee712f85b388f45c4d602"} Jul 15 13:36:51.196269 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:51.196246 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:51.304257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:51.304224 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a22eacc-89bb-427b-aba2-68e9bf1d8f9c-maas-api-service-ca\") pod \"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c\" (UID: \"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c\") " Jul 15 13:36:51.304624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:51.304596 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a22eacc-89bb-427b-aba2-68e9bf1d8f9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a22eacc-89bb-427b-aba2-68e9bf1d8f9c" (UID: "5a22eacc-89bb-427b-aba2-68e9bf1d8f9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:36:51.405969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:51.405881 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a22eacc-89bb-427b-aba2-68e9bf1d8f9c-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:36:52.073340 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:52.073310 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" Jul 15 13:36:52.073340 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:52.073331 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs" event={"ID":"5a22eacc-89bb-427b-aba2-68e9bf1d8f9c","Type":"ContainerDied","Data":"d66ff6bb1fd24d90f1272e17e6428d9ad3a02c5714d6bc5169b56ee07569bebb"} Jul 15 13:36:52.073605 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:52.073377 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d66ff6bb1fd24d90f1272e17e6428d9ad3a02c5714d6bc5169b56ee07569bebb" Jul 15 13:36:57.922637 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:57.922604 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq"] Jul 15 13:36:57.923074 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:36:57.922792 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" containerID="cri-o://359a467f5489819d95cde59899ffe3ca7b2e7d0b96169d53f8892db2c59655f7" gracePeriod=30 Jul 15 13:37:00.377350 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:00.377310 2564 patch_prober.go:28] interesting pod/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 13:37:00.377782 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:00.377373 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:37:03.131710 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.131681 2564 generic.go:356] "Generic (PLEG): container finished" podID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerID="359a467f5489819d95cde59899ffe3ca7b2e7d0b96169d53f8892db2c59655f7" exitCode=0 Jul 15 13:37:03.132034 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.131737 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" event={"ID":"0c414ea1-b32e-4342-bb7b-a1f28300b10c","Type":"ContainerDied","Data":"359a467f5489819d95cde59899ffe3ca7b2e7d0b96169d53f8892db2c59655f7"} Jul 15 13:37:03.132034 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.131784 2564 scope.go:117] "RemoveContainer" containerID="3cd061ad43854985261e36ef9f54927d868a03c6341b0e375b4024c2a5380b8d" Jul 15 13:37:03.172366 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.172343 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:37:03.212168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212108 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-data\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212141 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-credential-socket\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212176 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njfhh\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-kube-api-access-njfhh\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212220 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-certs\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212269 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-socket\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istiod-ca-cert\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212347 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212330 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-token\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212615 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212379 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-envoy\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212615 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212403 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:03.212615 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212437 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-podinfo\") pod \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\" (UID: \"0c414ea1-b32e-4342-bb7b-a1f28300b10c\") " Jul 15 13:37:03.212755 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212711 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:37:03.212828 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212811 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.212879 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212833 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.213162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.212998 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:03.213162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.213077 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:03.213162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.213098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:03.214967 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.214935 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-kube-api-access-njfhh" (OuterVolumeSpecName: "kube-api-access-njfhh") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "kube-api-access-njfhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:37:03.215065 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.215011 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:37:03.215125 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.215106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:37:03.215221 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.215202 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c414ea1-b32e-4342-bb7b-a1f28300b10c" (UID: "0c414ea1-b32e-4342-bb7b-a1f28300b10c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:37:03.314269 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314239 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.314269 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314264 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.314404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314273 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.314404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314282 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.314404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314291 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.314404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314298 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c414ea1-b32e-4342-bb7b-a1f28300b10c-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:03.314404 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:03.314306 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njfhh\" (UniqueName: \"kubernetes.io/projected/0c414ea1-b32e-4342-bb7b-a1f28300b10c-kube-api-access-njfhh\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:04.138402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.138372 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" Jul 15 13:37:04.138402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.138392 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq" event={"ID":"0c414ea1-b32e-4342-bb7b-a1f28300b10c","Type":"ContainerDied","Data":"6d9c22a7f01cc57dff7e6cd0d00a1b4d7641fea1d3014fe4bf7c4e78979f7a4a"} Jul 15 13:37:04.138937 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.138449 2564 scope.go:117] "RemoveContainer" containerID="359a467f5489819d95cde59899ffe3ca7b2e7d0b96169d53f8892db2c59655f7" Jul 15 13:37:04.160881 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.160852 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq"] Jul 15 13:37:04.164934 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.164912 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-faa562cb-gw-openshift-default-88c5df97f-qg2tq"] Jul 15 13:37:04.354460 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354432 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr"] Jul 15 13:37:04.354928 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354901 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" Jul 15 13:37:04.354928 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354928 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" Jul 15 13:37:04.355084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354946 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a22eacc-89bb-427b-aba2-68e9bf1d8f9c" containerName="revoke-keys" Jul 15 13:37:04.355084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354955 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a22eacc-89bb-427b-aba2-68e9bf1d8f9c" containerName="revoke-keys" Jul 15 13:37:04.355084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354974 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" Jul 15 13:37:04.355084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354980 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" Jul 15 13:37:04.355084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.354998 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:37:04.355084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.355004 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:37:04.355286 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.355093 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" Jul 15 13:37:04.355286 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.355103 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a22eacc-89bb-427b-aba2-68e9bf1d8f9c" containerName="revoke-keys" Jul 15 13:37:04.355286 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.355110 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="90770255-ef0c-469a-9081-e4a2b9a278cc" containerName="revoke-keys" Jul 15 13:37:04.355286 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.355117 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" containerName="istio-proxy" Jul 15 13:37:04.359353 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.359337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.361753 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.361732 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b9daa40c-openshift-default-dockercfg-22l2c\"" Jul 15 13:37:04.361884 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.361737 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:37:04.370624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.370604 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr"] Jul 15 13:37:04.423574 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423516 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-envoy\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423591 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkm67\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-kube-api-access-wkm67\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423630 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-certs\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istiod-ca-cert\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423813 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-token\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423813 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423722 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-podinfo\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423813 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-credential-socket\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-data\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.423919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.423912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-socket\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525137 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525108 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-envoy\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525146 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkm67\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-kube-api-access-wkm67\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-certs\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525216 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istiod-ca-cert\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-token\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525284 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-podinfo\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525604 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-credential-socket\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525604 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-data\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525604 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-socket\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525772 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525645 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-certs\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525772 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-credential-socket\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525772 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-data\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.525919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525796 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-socket\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.526005 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.525986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istiod-ca-cert\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.527979 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.527950 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-envoy\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.535288 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.535269 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-token\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.535396 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.535323 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-podinfo\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.536285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.536267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkm67\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-kube-api-access-wkm67\") pod \"e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.658051 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.658020 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c414ea1-b32e-4342-bb7b-a1f28300b10c" path="/var/lib/kubelet/pods/0c414ea1-b32e-4342-bb7b-a1f28300b10c/volumes" Jul 15 13:37:04.671828 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.671808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:04.806024 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.805993 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr"] Jul 15 13:37:04.807634 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:37:04.807610 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod438216a0_f9ac_4e18_a2dc_ffc713db9cd9.slice/crio-4d8b266127d4fafc860efa04e51c3e0454ec6c2707c150934b58ce5000c4662d WatchSource:0}: Error finding container 4d8b266127d4fafc860efa04e51c3e0454ec6c2707c150934b58ce5000c4662d: Status 404 returned error can't find the container with id 4d8b266127d4fafc860efa04e51c3e0454ec6c2707c150934b58ce5000c4662d Jul 15 13:37:04.809763 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.809734 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:37:04.809829 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.809799 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:37:04.809829 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:04.809826 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:37:05.144971 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:05.144930 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" event={"ID":"438216a0-f9ac-4e18-a2dc-ffc713db9cd9","Type":"ContainerStarted","Data":"912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080"} Jul 15 13:37:05.145407 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:05.144991 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" event={"ID":"438216a0-f9ac-4e18-a2dc-ffc713db9cd9","Type":"ContainerStarted","Data":"4d8b266127d4fafc860efa04e51c3e0454ec6c2707c150934b58ce5000c4662d"} Jul 15 13:37:05.169565 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:05.169515 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" podStartSLOduration=1.169499385 podStartE2EDuration="1.169499385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:37:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:37:05.168238217 +0000 UTC m=+1831.123692802" watchObservedRunningTime="2026-07-15 13:37:05.169499385 +0000 UTC m=+1831.124953946" Jul 15 13:37:05.673003 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:05.672974 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:05.677673 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:05.677648 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:06.151081 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:06.151050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:06.151988 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:06.151962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:10.583304 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.583267 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w"] Jul 15 13:37:10.586975 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.586956 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:10.589187 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.589158 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:37:10.589306 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.589229 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:37:10.595530 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.595506 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w"] Jul 15 13:37:10.681721 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.681689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w\" (UID: \"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:10.783218 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.783184 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w\" (UID: \"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:10.783849 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.783830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w\" (UID: \"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:10.898845 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:10.898765 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:11.289194 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:11.289171 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w"] Jul 15 13:37:11.291453 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:37:11.291427 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d8c9e8b_0be4_4106_a0cd_0a5d885bc20c.slice/crio-08bda10d6cfc054db1187493f90b61b652620d1f65755850a33df2a912029277 WatchSource:0}: Error finding container 08bda10d6cfc054db1187493f90b61b652620d1f65755850a33df2a912029277: Status 404 returned error can't find the container with id 08bda10d6cfc054db1187493f90b61b652620d1f65755850a33df2a912029277 Jul 15 13:37:12.184176 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:12.184147 2564 generic.go:356] "Generic (PLEG): container finished" podID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerID="14816c17ffe628d5d924a17ef82a0e46ac1482d520599e1960493223e5975561" exitCode=7 Jul 15 13:37:12.184454 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:12.184234 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" event={"ID":"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c","Type":"ContainerDied","Data":"14816c17ffe628d5d924a17ef82a0e46ac1482d520599e1960493223e5975561"} Jul 15 13:37:12.184454 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:12.184287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" event={"ID":"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c","Type":"ContainerStarted","Data":"08bda10d6cfc054db1187493f90b61b652620d1f65755850a33df2a912029277"} Jul 15 13:37:12.184454 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:12.184411 2564 scope.go:117] "RemoveContainer" containerID="14816c17ffe628d5d924a17ef82a0e46ac1482d520599e1960493223e5975561" Jul 15 13:37:13.191574 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:13.191547 2564 generic.go:356] "Generic (PLEG): container finished" podID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerID="406844e25f426d448e1aa80237ffcaf9c4735fe02efc7f72cf2e4d25aecb4495" exitCode=7 Jul 15 13:37:13.191871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:13.191636 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" event={"ID":"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c","Type":"ContainerDied","Data":"406844e25f426d448e1aa80237ffcaf9c4735fe02efc7f72cf2e4d25aecb4495"} Jul 15 13:37:13.191871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:13.191691 2564 scope.go:117] "RemoveContainer" containerID="14816c17ffe628d5d924a17ef82a0e46ac1482d520599e1960493223e5975561" Jul 15 13:37:13.191871 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:13.191850 2564 scope.go:117] "RemoveContainer" containerID="406844e25f426d448e1aa80237ffcaf9c4735fe02efc7f72cf2e4d25aecb4495" Jul 15 13:37:13.192105 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:37:13.192085 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w_odh-ai-gateway-infra(2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" Jul 15 13:37:14.199158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:14.199137 2564 scope.go:117] "RemoveContainer" containerID="406844e25f426d448e1aa80237ffcaf9c4735fe02efc7f72cf2e4d25aecb4495" Jul 15 13:37:14.199562 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:37:14.199353 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w_odh-ai-gateway-infra(2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" Jul 15 13:37:17.216708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:17.216685 2564 generic.go:356] "Generic (PLEG): container finished" podID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerID="912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080" exitCode=0 Jul 15 13:37:17.217037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:17.216762 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" event={"ID":"438216a0-f9ac-4e18-a2dc-ffc713db9cd9","Type":"ContainerDied","Data":"912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080"} Jul 15 13:37:17.217148 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:17.217135 2564 scope.go:117] "RemoveContainer" containerID="912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080" Jul 15 13:37:17.218158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:17.218130 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:37:17.218250 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:17.218221 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:37:17.218312 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:17.218281 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:37:18.223227 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:18.223195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" event={"ID":"438216a0-f9ac-4e18-a2dc-ffc713db9cd9","Type":"ContainerStarted","Data":"1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba"} Jul 15 13:37:18.672621 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:18.672594 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:18.673733 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:18.673706 2564 patch_prober.go:28] interesting pod/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:37:18.673876 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:18.673760 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:37:19.672856 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:19.672817 2564 patch_prober.go:28] interesting pod/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:37:19.673239 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:19.672873 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:37:20.673148 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:20.673111 2564 patch_prober.go:28] interesting pod/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:37:20.673560 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:20.673162 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:37:21.151881 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:21.151832 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:21.672406 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:21.672361 2564 patch_prober.go:28] interesting pod/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Jul 15 13:37:21.672613 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:21.672427 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Jul 15 13:37:22.676547 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:22.676514 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:22.677339 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:22.677318 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:37:26.651759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:26.651725 2564 scope.go:117] "RemoveContainer" containerID="406844e25f426d448e1aa80237ffcaf9c4735fe02efc7f72cf2e4d25aecb4495" Jul 15 13:37:27.299042 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:27.299012 2564 generic.go:356] "Generic (PLEG): container finished" podID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerID="c41e67c3964c12ac781269972d38a5a0996f0409a2786710e2bdb8cc4177c1e9" exitCode=0 Jul 15 13:37:27.299165 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:27.299098 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" event={"ID":"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c","Type":"ContainerDied","Data":"c41e67c3964c12ac781269972d38a5a0996f0409a2786710e2bdb8cc4177c1e9"} Jul 15 13:37:27.299165 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:27.299159 2564 scope.go:117] "RemoveContainer" containerID="406844e25f426d448e1aa80237ffcaf9c4735fe02efc7f72cf2e4d25aecb4495" Jul 15 13:37:28.333183 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:28.333146 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w"] Jul 15 13:37:28.441144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:28.441119 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:28.465402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:28.465361 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c-maas-api-service-ca\") pod \"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c\" (UID: \"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c\") " Jul 15 13:37:28.467093 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:28.467066 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" (UID: "2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:37:28.567898 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:28.567819 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:37:29.312774 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:29.312744 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" Jul 15 13:37:29.312943 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:29.312788 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w" event={"ID":"2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c","Type":"ContainerDied","Data":"08bda10d6cfc054db1187493f90b61b652620d1f65755850a33df2a912029277"} Jul 15 13:37:29.312943 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:29.312840 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08bda10d6cfc054db1187493f90b61b652620d1f65755850a33df2a912029277" Jul 15 13:37:29.338825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:29.338796 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w"] Jul 15 13:37:29.342353 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:29.342329 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-gc89w"] Jul 15 13:37:30.656762 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:37:30.656727 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" path="/var/lib/kubelet/pods/2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c/volumes" Jul 15 13:38:00.626148 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626065 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m"] Jul 15 13:38:00.626594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626506 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.626594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626519 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.626594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626534 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.626594 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626539 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.626744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626609 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.626744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626619 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.626744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.626627 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:00.629761 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.629746 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:00.632159 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.632137 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:38:00.632159 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.632150 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:38:00.636734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.636709 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m"] Jul 15 13:38:00.751163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.751126 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2767e48-be80-4a67-b613-9b189ed13d9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m\" (UID: \"d2767e48-be80-4a67-b613-9b189ed13d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:00.851909 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.851877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2767e48-be80-4a67-b613-9b189ed13d9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m\" (UID: \"d2767e48-be80-4a67-b613-9b189ed13d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:00.852530 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.852512 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2767e48-be80-4a67-b613-9b189ed13d9c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m\" (UID: \"d2767e48-be80-4a67-b613-9b189ed13d9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:00.941966 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:00.941870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:01.074361 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:01.074336 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m"] Jul 15 13:38:01.076736 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:38:01.076704 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2767e48_be80_4a67_b613_9b189ed13d9c.slice/crio-e9b99571eaf7187c43c85ab12a2b89d9a8f663d7a73f22167507a6650cad93d1 WatchSource:0}: Error finding container e9b99571eaf7187c43c85ab12a2b89d9a8f663d7a73f22167507a6650cad93d1: Status 404 returned error can't find the container with id e9b99571eaf7187c43c85ab12a2b89d9a8f663d7a73f22167507a6650cad93d1 Jul 15 13:38:01.483199 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:01.483163 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" event={"ID":"d2767e48-be80-4a67-b613-9b189ed13d9c","Type":"ContainerStarted","Data":"c333c57f8a936aaf671dbb5cb12cc0ade91176c73aa8a1faa5648bb7fde765a0"} Jul 15 13:38:01.483375 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:01.483229 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" event={"ID":"d2767e48-be80-4a67-b613-9b189ed13d9c","Type":"ContainerStarted","Data":"e9b99571eaf7187c43c85ab12a2b89d9a8f663d7a73f22167507a6650cad93d1"} Jul 15 13:38:01.500015 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:01.499961 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" podStartSLOduration=1.49994512 podStartE2EDuration="1.49994512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:38:01.498291056 +0000 UTC m=+1887.453745626" watchObservedRunningTime="2026-07-15 13:38:01.49994512 +0000 UTC m=+1887.455399682" Jul 15 13:38:02.489764 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:02.489730 2564 generic.go:356] "Generic (PLEG): container finished" podID="d2767e48-be80-4a67-b613-9b189ed13d9c" containerID="c333c57f8a936aaf671dbb5cb12cc0ade91176c73aa8a1faa5648bb7fde765a0" exitCode=0 Jul 15 13:38:02.490145 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:02.489802 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" event={"ID":"d2767e48-be80-4a67-b613-9b189ed13d9c","Type":"ContainerDied","Data":"c333c57f8a936aaf671dbb5cb12cc0ade91176c73aa8a1faa5648bb7fde765a0"} Jul 15 13:38:03.628619 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:03.628593 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:03.781038 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:03.780948 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2767e48-be80-4a67-b613-9b189ed13d9c-maas-api-service-ca\") pod \"d2767e48-be80-4a67-b613-9b189ed13d9c\" (UID: \"d2767e48-be80-4a67-b613-9b189ed13d9c\") " Jul 15 13:38:03.781297 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:03.781274 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2767e48-be80-4a67-b613-9b189ed13d9c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2767e48-be80-4a67-b613-9b189ed13d9c" (UID: "d2767e48-be80-4a67-b613-9b189ed13d9c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:38:03.882275 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:03.882239 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2767e48-be80-4a67-b613-9b189ed13d9c-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:04.508081 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:04.508052 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" Jul 15 13:38:04.508274 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:04.508101 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m" event={"ID":"d2767e48-be80-4a67-b613-9b189ed13d9c","Type":"ContainerDied","Data":"e9b99571eaf7187c43c85ab12a2b89d9a8f663d7a73f22167507a6650cad93d1"} Jul 15 13:38:04.508274 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:04.508150 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b99571eaf7187c43c85ab12a2b89d9a8f663d7a73f22167507a6650cad93d1" Jul 15 13:38:26.704317 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.704283 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr"] Jul 15 13:38:26.704803 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.704542 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" containerID="cri-o://1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba" gracePeriod=30 Jul 15 13:38:26.984985 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.984902 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc"] Jul 15 13:38:26.985371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.985357 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:26.985419 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.985373 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d8c9e8b-0be4-4106-a0cd-0a5d885bc20c" containerName="revoke-keys" Jul 15 13:38:26.985419 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.985382 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2767e48-be80-4a67-b613-9b189ed13d9c" containerName="revoke-keys" Jul 15 13:38:26.985419 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.985388 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2767e48-be80-4a67-b613-9b189ed13d9c" containerName="revoke-keys" Jul 15 13:38:26.985558 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.985542 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2767e48-be80-4a67-b613-9b189ed13d9c" containerName="revoke-keys" Jul 15 13:38:26.988527 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.988510 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:26.990985 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.990958 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-c15b0ffb-openshift-default-dockercfg-7qqt6\"" Jul 15 13:38:26.998501 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:26.998461 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc"] Jul 15 13:38:27.091320 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-certs\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091512 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091345 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-envoy\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091512 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-credential-socket\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091512 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091409 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-socket\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091714 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091564 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-token\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091714 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091602 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-data\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091714 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091679 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rvjt\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-kube-api-access-2rvjt\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091906 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istiod-ca-cert\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.091906 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.091745 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-podinfo\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.192701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192668 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-certs\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.192838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192716 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-envoy\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.192838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192749 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-credential-socket\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.192838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-socket\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.192838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192808 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-token\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.192838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-data\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rvjt\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-kube-api-access-2rvjt\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istiod-ca-cert\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.192911 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-podinfo\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.193085 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-certs\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.193133 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-credential-socket\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193323 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.193260 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-socket\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193357 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.193325 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-data\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.193609 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.193589 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istiod-ca-cert\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.195164 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.195143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-envoy\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.195400 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.195379 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-podinfo\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.200852 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.200833 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-token\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.201162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.201142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rvjt\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-kube-api-access-2rvjt\") pod \"e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.301167 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.301144 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:27.434116 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.434088 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc"] Jul 15 13:38:27.435566 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:38:27.435537 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72a90e41_5926_4c0a_b101_0a9aeee8c9d3.slice/crio-d580952d3d5cf002ccd11bd73a3cbee4aed287a806827e2a13434fd3840a4231 WatchSource:0}: Error finding container d580952d3d5cf002ccd11bd73a3cbee4aed287a806827e2a13434fd3840a4231: Status 404 returned error can't find the container with id d580952d3d5cf002ccd11bd73a3cbee4aed287a806827e2a13434fd3840a4231 Jul 15 13:38:27.437957 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.437925 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:38:27.438042 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.437992 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:38:27.438042 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.438017 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:38:27.636322 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.636222 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" event={"ID":"72a90e41-5926-4c0a-b101-0a9aeee8c9d3","Type":"ContainerStarted","Data":"362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb"} Jul 15 13:38:27.636322 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.636289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" event={"ID":"72a90e41-5926-4c0a-b101-0a9aeee8c9d3","Type":"ContainerStarted","Data":"d580952d3d5cf002ccd11bd73a3cbee4aed287a806827e2a13434fd3840a4231"} Jul 15 13:38:27.656264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:27.656216 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podStartSLOduration=1.6562001720000001 podStartE2EDuration="1.656200172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:38:27.654934666 +0000 UTC m=+1913.610389233" watchObservedRunningTime="2026-07-15 13:38:27.656200172 +0000 UTC m=+1913.611654736" Jul 15 13:38:28.302033 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:28.301997 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:28.307793 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:28.307767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:28.648760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:28.648681 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:28.649635 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:28.649612 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:31.997828 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:31.997804 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.153795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-data\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.153860 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-certs\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.153905 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-socket\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.153948 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-credential-socket\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.154032 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-podinfo\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.154067 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-envoy\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.154103 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istiod-ca-cert\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.154143 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-token\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.154929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.154192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wkm67\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-kube-api-access-wkm67\") pod \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\" (UID: \"438216a0-f9ac-4e18-a2dc-ffc713db9cd9\") " Jul 15 13:38:32.155993 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.155762 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:38:32.155993 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.155949 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:38:32.155993 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.155957 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:38:32.156490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.156428 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:38:32.156618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.156529 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-data" (OuterVolumeSpecName: "istio-data") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:38:32.157104 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.157079 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-kube-api-access-wkm67" (OuterVolumeSpecName: "kube-api-access-wkm67") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "kube-api-access-wkm67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:38:32.157916 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.157891 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:38:32.158221 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.158200 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-token" (OuterVolumeSpecName: "istio-token") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:38:32.158559 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.158541 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "438216a0-f9ac-4e18-a2dc-ffc713db9cd9" (UID: "438216a0-f9ac-4e18-a2dc-ffc713db9cd9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.255937 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkm67\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-kube-api-access-wkm67\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.255975 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.255985 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.255992 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.256001 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.256011 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.256019 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.256027 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.256031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.256035 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/438216a0-f9ac-4e18-a2dc-ffc713db9cd9-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:32.530174 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530090 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x"] Jul 15 13:38:32.530580 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530559 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" Jul 15 13:38:32.530674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530582 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" Jul 15 13:38:32.530674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530611 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" Jul 15 13:38:32.530674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530620 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" Jul 15 13:38:32.530833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530733 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" Jul 15 13:38:32.530833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.530752 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerName="istio-proxy" Jul 15 13:38:32.535821 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.535793 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:32.538410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.538381 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:38:32.538705 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.538683 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:38:32.542968 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.542929 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x"] Jul 15 13:38:32.660598 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.660558 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295d21c0-b95d-4ebd-98bd-27c8f3096ba3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x\" (UID: \"295d21c0-b95d-4ebd-98bd-27c8f3096ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:32.670324 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.670297 2564 generic.go:356] "Generic (PLEG): container finished" podID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" containerID="1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba" exitCode=0 Jul 15 13:38:32.670428 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.670344 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" Jul 15 13:38:32.670428 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.670381 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" event={"ID":"438216a0-f9ac-4e18-a2dc-ffc713db9cd9","Type":"ContainerDied","Data":"1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba"} Jul 15 13:38:32.670569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.670446 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr" event={"ID":"438216a0-f9ac-4e18-a2dc-ffc713db9cd9","Type":"ContainerDied","Data":"4d8b266127d4fafc860efa04e51c3e0454ec6c2707c150934b58ce5000c4662d"} Jul 15 13:38:32.670569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.670492 2564 scope.go:117] "RemoveContainer" containerID="1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba" Jul 15 13:38:32.679270 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.679254 2564 scope.go:117] "RemoveContainer" containerID="912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080" Jul 15 13:38:32.687558 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.687544 2564 scope.go:117] "RemoveContainer" containerID="1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba" Jul 15 13:38:32.687624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.687601 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr"] Jul 15 13:38:32.687822 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:38:32.687796 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba\": container with ID starting with 1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba not found: ID does not exist" containerID="1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba" Jul 15 13:38:32.687932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.687822 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba"} err="failed to get container status \"1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba\": rpc error: code = NotFound desc = could not find container \"1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba\": container with ID starting with 1377d4aa6f6b25a5cb5efc6b31c7e93f9b404f7deca835b22cafe5627b841cba not found: ID does not exist" Jul 15 13:38:32.687932 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.687840 2564 scope.go:117] "RemoveContainer" containerID="912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080" Jul 15 13:38:32.688115 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:38:32.688098 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080\": container with ID starting with 912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080 not found: ID does not exist" containerID="912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080" Jul 15 13:38:32.688155 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.688123 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080"} err="failed to get container status \"912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080\": rpc error: code = NotFound desc = could not find container \"912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080\": container with ID starting with 912c1f7491376079069a19e67f2e835d6b9a1868bc7b042c2838dc7ab2973080 not found: ID does not exist" Jul 15 13:38:32.691280 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.691258 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b9daa40c-openshift-default-c6b9f6d7-crttr"] Jul 15 13:38:32.761115 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.761074 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295d21c0-b95d-4ebd-98bd-27c8f3096ba3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x\" (UID: \"295d21c0-b95d-4ebd-98bd-27c8f3096ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:32.761788 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.761761 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295d21c0-b95d-4ebd-98bd-27c8f3096ba3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x\" (UID: \"295d21c0-b95d-4ebd-98bd-27c8f3096ba3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:32.856163 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:32.856130 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:33.006902 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:33.006869 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x"] Jul 15 13:38:33.009416 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:38:33.009370 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod295d21c0_b95d_4ebd_98bd_27c8f3096ba3.slice/crio-1e3323942a139f274d5b505ae044a061d07c43bc5615683a7b2c535f9e34a238 WatchSource:0}: Error finding container 1e3323942a139f274d5b505ae044a061d07c43bc5615683a7b2c535f9e34a238: Status 404 returned error can't find the container with id 1e3323942a139f274d5b505ae044a061d07c43bc5615683a7b2c535f9e34a238 Jul 15 13:38:33.683812 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:33.683775 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" event={"ID":"295d21c0-b95d-4ebd-98bd-27c8f3096ba3","Type":"ContainerStarted","Data":"6ed5f7468a620fd31d5b6ce2206ac9ab1a58678d45e132d6835d7382b3459e9e"} Jul 15 13:38:33.684031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:33.683868 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" event={"ID":"295d21c0-b95d-4ebd-98bd-27c8f3096ba3","Type":"ContainerStarted","Data":"1e3323942a139f274d5b505ae044a061d07c43bc5615683a7b2c535f9e34a238"} Jul 15 13:38:33.699639 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:33.699583 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" podStartSLOduration=1.69953512 podStartE2EDuration="1.69953512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:38:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:38:33.698679787 +0000 UTC m=+1919.654134350" watchObservedRunningTime="2026-07-15 13:38:33.69953512 +0000 UTC m=+1919.654989685" Jul 15 13:38:34.663696 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:34.663664 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="438216a0-f9ac-4e18-a2dc-ffc713db9cd9" path="/var/lib/kubelet/pods/438216a0-f9ac-4e18-a2dc-ffc713db9cd9/volumes" Jul 15 13:38:34.691786 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:34.691749 2564 generic.go:356] "Generic (PLEG): container finished" podID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerID="6ed5f7468a620fd31d5b6ce2206ac9ab1a58678d45e132d6835d7382b3459e9e" exitCode=7 Jul 15 13:38:34.691928 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:34.691834 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" event={"ID":"295d21c0-b95d-4ebd-98bd-27c8f3096ba3","Type":"ContainerDied","Data":"6ed5f7468a620fd31d5b6ce2206ac9ab1a58678d45e132d6835d7382b3459e9e"} Jul 15 13:38:34.692031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:34.692018 2564 scope.go:117] "RemoveContainer" containerID="6ed5f7468a620fd31d5b6ce2206ac9ab1a58678d45e132d6835d7382b3459e9e" Jul 15 13:38:35.699754 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:35.699720 2564 generic.go:356] "Generic (PLEG): container finished" podID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerID="2d61e80384f4bc826c7953e125584706e5215016be38157aab86ebdf98ae3166" exitCode=7 Jul 15 13:38:35.700218 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:35.699797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" event={"ID":"295d21c0-b95d-4ebd-98bd-27c8f3096ba3","Type":"ContainerDied","Data":"2d61e80384f4bc826c7953e125584706e5215016be38157aab86ebdf98ae3166"} Jul 15 13:38:35.700218 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:35.699855 2564 scope.go:117] "RemoveContainer" containerID="6ed5f7468a620fd31d5b6ce2206ac9ab1a58678d45e132d6835d7382b3459e9e" Jul 15 13:38:35.700218 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:35.700054 2564 scope.go:117] "RemoveContainer" containerID="2d61e80384f4bc826c7953e125584706e5215016be38157aab86ebdf98ae3166" Jul 15 13:38:35.700370 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:38:35.700322 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x_odh-ai-gateway-infra(295d21c0-b95d-4ebd-98bd-27c8f3096ba3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" Jul 15 13:38:36.707727 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:36.707645 2564 scope.go:117] "RemoveContainer" containerID="2d61e80384f4bc826c7953e125584706e5215016be38157aab86ebdf98ae3166" Jul 15 13:38:36.708101 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:38:36.707846 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x_odh-ai-gateway-infra(295d21c0-b95d-4ebd-98bd-27c8f3096ba3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" Jul 15 13:38:38.726617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:38.726581 2564 generic.go:356] "Generic (PLEG): container finished" podID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerID="362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb" exitCode=0 Jul 15 13:38:38.726999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:38.726654 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" event={"ID":"72a90e41-5926-4c0a-b101-0a9aeee8c9d3","Type":"ContainerDied","Data":"362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb"} Jul 15 13:38:38.727170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:38.727154 2564 scope.go:117] "RemoveContainer" containerID="362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb" Jul 15 13:38:38.728204 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:38.728175 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:38:38.728324 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:38.728251 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:38:38.728324 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:38.728296 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:38:39.756312 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:39.756272 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" event={"ID":"72a90e41-5926-4c0a-b101-0a9aeee8c9d3","Type":"ContainerStarted","Data":"93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41"} Jul 15 13:38:40.301895 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:40.301859 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:40.302991 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:40.302964 2564 patch_prober.go:28] interesting pod/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 13:38:40.303104 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:40.303010 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 13:38:41.302328 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:41.302295 2564 patch_prober.go:28] interesting pod/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 13:38:41.302729 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:41.302348 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 13:38:42.302295 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:42.302260 2564 patch_prober.go:28] interesting pod/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 13:38:42.302505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:42.302323 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 13:38:43.302597 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:43.302563 2564 patch_prober.go:28] interesting pod/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Jul 15 13:38:43.303016 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:43.302617 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Jul 15 13:38:43.648958 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:43.648878 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:44.305957 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:44.305926 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:44.306899 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:44.306877 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:38:49.651919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:49.651890 2564 scope.go:117] "RemoveContainer" containerID="2d61e80384f4bc826c7953e125584706e5215016be38157aab86ebdf98ae3166" Jul 15 13:38:50.817225 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:50.817187 2564 generic.go:356] "Generic (PLEG): container finished" podID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerID="6ba158b830ddd6dd0802c1067d60bc19a4aa5e5cd2b55b8ed8558387af81d1bb" exitCode=0 Jul 15 13:38:50.817706 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:50.817254 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" event={"ID":"295d21c0-b95d-4ebd-98bd-27c8f3096ba3","Type":"ContainerDied","Data":"6ba158b830ddd6dd0802c1067d60bc19a4aa5e5cd2b55b8ed8558387af81d1bb"} Jul 15 13:38:50.817706 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:50.817322 2564 scope.go:117] "RemoveContainer" containerID="2d61e80384f4bc826c7953e125584706e5215016be38157aab86ebdf98ae3166" Jul 15 13:38:51.846035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:51.845998 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x"] Jul 15 13:38:51.958927 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:51.958900 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:52.055400 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.055363 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295d21c0-b95d-4ebd-98bd-27c8f3096ba3-maas-api-service-ca\") pod \"295d21c0-b95d-4ebd-98bd-27c8f3096ba3\" (UID: \"295d21c0-b95d-4ebd-98bd-27c8f3096ba3\") " Jul 15 13:38:52.055801 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.055777 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/295d21c0-b95d-4ebd-98bd-27c8f3096ba3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "295d21c0-b95d-4ebd-98bd-27c8f3096ba3" (UID: "295d21c0-b95d-4ebd-98bd-27c8f3096ba3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:38:52.157238 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.157159 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/295d21c0-b95d-4ebd-98bd-27c8f3096ba3-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:38:52.830424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.830395 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" Jul 15 13:38:52.830424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.830410 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x" event={"ID":"295d21c0-b95d-4ebd-98bd-27c8f3096ba3","Type":"ContainerDied","Data":"1e3323942a139f274d5b505ae044a061d07c43bc5615683a7b2c535f9e34a238"} Jul 15 13:38:52.830667 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.830449 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e3323942a139f274d5b505ae044a061d07c43bc5615683a7b2c535f9e34a238" Jul 15 13:38:52.867235 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.867205 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x"] Jul 15 13:38:52.871429 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:52.871406 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-x5j2x"] Jul 15 13:38:54.657380 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:38:54.657348 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" path="/var/lib/kubelet/pods/295d21c0-b95d-4ebd-98bd-27c8f3096ba3/volumes" Jul 15 13:39:32.578677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.578647 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc"] Jul 15 13:39:32.579087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.579060 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:39:32.579144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.579103 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:39:32.579144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.579127 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:39:32.579144 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.579135 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:39:32.579277 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.579262 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:39:32.579312 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.579280 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:39:32.581496 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.581464 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:32.584010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.583989 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:39:32.584096 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.583989 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:39:32.590503 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.590481 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc"] Jul 15 13:39:32.613753 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.613727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c56b591-8cd7-4b08-add3-6348acf42b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc\" (UID: \"1c56b591-8cd7-4b08-add3-6348acf42b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:32.714533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.714506 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c56b591-8cd7-4b08-add3-6348acf42b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc\" (UID: \"1c56b591-8cd7-4b08-add3-6348acf42b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:32.715071 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.715053 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c56b591-8cd7-4b08-add3-6348acf42b77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc\" (UID: \"1c56b591-8cd7-4b08-add3-6348acf42b77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:32.892864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:32.892787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:33.233731 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:33.233705 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc"] Jul 15 13:39:33.235413 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:39:33.235386 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c56b591_8cd7_4b08_add3_6348acf42b77.slice/crio-c0c6ba0bbb2d51a36c90eb489b5e98793545d8a5795b15b83a7d4991662d6064 WatchSource:0}: Error finding container c0c6ba0bbb2d51a36c90eb489b5e98793545d8a5795b15b83a7d4991662d6064: Status 404 returned error can't find the container with id c0c6ba0bbb2d51a36c90eb489b5e98793545d8a5795b15b83a7d4991662d6064 Jul 15 13:39:34.052316 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:34.052281 2564 generic.go:356] "Generic (PLEG): container finished" podID="1c56b591-8cd7-4b08-add3-6348acf42b77" containerID="cd15d0fcb187fd4d97465f714e531d1b9cadf2d03d039abb8357a63fd06a6043" exitCode=0 Jul 15 13:39:34.052688 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:34.052337 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" event={"ID":"1c56b591-8cd7-4b08-add3-6348acf42b77","Type":"ContainerDied","Data":"cd15d0fcb187fd4d97465f714e531d1b9cadf2d03d039abb8357a63fd06a6043"} Jul 15 13:39:34.052688 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:34.052421 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" event={"ID":"1c56b591-8cd7-4b08-add3-6348acf42b77","Type":"ContainerStarted","Data":"c0c6ba0bbb2d51a36c90eb489b5e98793545d8a5795b15b83a7d4991662d6064"} Jul 15 13:39:35.185054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:35.185029 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:35.238235 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:35.238210 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c56b591-8cd7-4b08-add3-6348acf42b77-maas-api-service-ca\") pod \"1c56b591-8cd7-4b08-add3-6348acf42b77\" (UID: \"1c56b591-8cd7-4b08-add3-6348acf42b77\") " Jul 15 13:39:35.238562 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:35.238543 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c56b591-8cd7-4b08-add3-6348acf42b77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1c56b591-8cd7-4b08-add3-6348acf42b77" (UID: "1c56b591-8cd7-4b08-add3-6348acf42b77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:39:35.339691 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:35.339633 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c56b591-8cd7-4b08-add3-6348acf42b77-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:39:36.067307 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:36.067278 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" Jul 15 13:39:36.067307 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:36.067284 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc" event={"ID":"1c56b591-8cd7-4b08-add3-6348acf42b77","Type":"ContainerDied","Data":"c0c6ba0bbb2d51a36c90eb489b5e98793545d8a5795b15b83a7d4991662d6064"} Jul 15 13:39:36.067595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:36.067328 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0c6ba0bbb2d51a36c90eb489b5e98793545d8a5795b15b83a7d4991662d6064" Jul 15 13:39:57.856558 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:57.856525 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc"] Jul 15 13:39:57.856933 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:57.856728 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" containerID="cri-o://93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41" gracePeriod=30 Jul 15 13:39:59.308796 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:59.308758 2564 patch_prober.go:28] interesting pod/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 13:39:59.309153 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:39:59.308824 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:40:03.115313 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.115285 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:40:03.220900 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.220866 2564 generic.go:356] "Generic (PLEG): container finished" podID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerID="93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41" exitCode=0 Jul 15 13:40:03.221083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.220927 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" Jul 15 13:40:03.221083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.220950 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" event={"ID":"72a90e41-5926-4c0a-b101-0a9aeee8c9d3","Type":"ContainerDied","Data":"93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41"} Jul 15 13:40:03.221083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.221018 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc" event={"ID":"72a90e41-5926-4c0a-b101-0a9aeee8c9d3","Type":"ContainerDied","Data":"d580952d3d5cf002ccd11bd73a3cbee4aed287a806827e2a13434fd3840a4231"} Jul 15 13:40:03.221083 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.221047 2564 scope.go:117] "RemoveContainer" containerID="93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41" Jul 15 13:40:03.234564 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.234544 2564 scope.go:117] "RemoveContainer" containerID="362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb" Jul 15 13:40:03.244988 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.244961 2564 scope.go:117] "RemoveContainer" containerID="93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41" Jul 15 13:40:03.245321 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:40:03.245300 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41\": container with ID starting with 93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41 not found: ID does not exist" containerID="93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41" Jul 15 13:40:03.245427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.245347 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41"} err="failed to get container status \"93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41\": rpc error: code = NotFound desc = could not find container \"93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41\": container with ID starting with 93bc53e4d17d0a59c6b6406f0e010e302d29896ec75618eee3b8bb3cffa50b41 not found: ID does not exist" Jul 15 13:40:03.245427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.245372 2564 scope.go:117] "RemoveContainer" containerID="362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb" Jul 15 13:40:03.245670 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:40:03.245649 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb\": container with ID starting with 362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb not found: ID does not exist" containerID="362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb" Jul 15 13:40:03.245760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.245675 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb"} err="failed to get container status \"362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb\": rpc error: code = NotFound desc = could not find container \"362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb\": container with ID starting with 362fe231ff5c659df9b7686a4994c25946af70e27c48c47b38a120947fbebfdb not found: ID does not exist" Jul 15 13:40:03.306244 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306215 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istiod-ca-cert\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-podinfo\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306285 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-envoy\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306318 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-data\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306345 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-certs\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306359 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-credential-socket\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306386 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2rvjt\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-kube-api-access-2rvjt\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306798 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306454 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-token\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306798 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306508 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-socket\") pod \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\" (UID: \"72a90e41-5926-4c0a-b101-0a9aeee8c9d3\") " Jul 15 13:40:03.306798 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306686 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:40:03.306798 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306685 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:40:03.307037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.306898 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:40:03.307037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.307005 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:40:03.307161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.307049 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.307161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.307071 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.307161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.307083 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.307317 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.307178 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-data" (OuterVolumeSpecName: "istio-data") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:40:03.308825 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.308799 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-kube-api-access-2rvjt" (OuterVolumeSpecName: "kube-api-access-2rvjt") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "kube-api-access-2rvjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:40:03.309141 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.309113 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:40:03.309233 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.309171 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:40:03.309233 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.309199 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-token" (OuterVolumeSpecName: "istio-token") pod "72a90e41-5926-4c0a-b101-0a9aeee8c9d3" (UID: "72a90e41-5926-4c0a-b101-0a9aeee8c9d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:40:03.408132 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.408095 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.408132 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.408132 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rvjt\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-kube-api-access-2rvjt\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.408272 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.408143 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.408272 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.408152 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.408272 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.408160 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.408272 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.408167 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/72a90e41-5926-4c0a-b101-0a9aeee8c9d3-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:40:03.543951 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.543924 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc"] Jul 15 13:40:03.549859 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:03.549837 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-c15b0ffb-openshift-default-86c974fccf-hgnhc"] Jul 15 13:40:04.658567 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:40:04.658485 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" path="/var/lib/kubelet/pods/72a90e41-5926-4c0a-b101-0a9aeee8c9d3/volumes" Jul 15 13:41:34.693334 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:34.693309 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:41:34.712017 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:34.711989 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:41:34.727671 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:34.727645 2564 scope.go:117] "RemoveContainer" containerID="b1199bb48640ce2d0feec28f139d1c6b934b606087edf936ea4263739a04e775" Jul 15 13:41:35.634081 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634050 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx"] Jul 15 13:41:35.634604 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634587 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c56b591-8cd7-4b08-add3-6348acf42b77" containerName="revoke-keys" Jul 15 13:41:35.634657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634606 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c56b591-8cd7-4b08-add3-6348acf42b77" containerName="revoke-keys" Jul 15 13:41:35.634657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634617 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" Jul 15 13:41:35.634657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634623 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" Jul 15 13:41:35.634657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634649 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:41:35.634657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634654 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:41:35.634818 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634717 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" Jul 15 13:41:35.634818 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634725 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" Jul 15 13:41:35.634818 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634731 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="295d21c0-b95d-4ebd-98bd-27c8f3096ba3" containerName="revoke-keys" Jul 15 13:41:35.634818 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.634739 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c56b591-8cd7-4b08-add3-6348acf42b77" containerName="revoke-keys" Jul 15 13:41:35.639305 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.639288 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.641773 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.641746 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:41:35.641897 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.641805 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b14f8bd8-openshift-default-dockercfg-bhqg9\"" Jul 15 13:41:35.654138 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.654115 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx"] Jul 15 13:41:35.655827 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.655806 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab63831-3790-43f8-8203-19424aa030d4-istiod-ca-cert\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.655962 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.655841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab63831-3790-43f8-8203-19424aa030d4-istio-podinfo\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.655962 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.655863 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-socket\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.655962 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.655911 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-data\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.656087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.655959 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-envoy\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.656087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.655990 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-credential-socket\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.656087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.656014 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-istio-token\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.656189 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.656097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-certs\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.656189 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.656140 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-kube-api-access-vq7qh\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757464 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757426 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-credential-socket\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-istio-token\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-certs\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-kube-api-access-vq7qh\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab63831-3790-43f8-8203-19424aa030d4-istiod-ca-cert\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab63831-3790-43f8-8203-19424aa030d4-istio-podinfo\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-socket\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-data\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-envoy\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.757969 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.757943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-credential-socket\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.758289 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.758035 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-certs\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.758289 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.758096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-data\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.758289 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.758118 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-socket\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.758394 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.758348 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab63831-3790-43f8-8203-19424aa030d4-istiod-ca-cert\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.760001 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.759975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-envoy\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.760206 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.760189 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab63831-3790-43f8-8203-19424aa030d4-istio-podinfo\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.765707 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.765685 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-istio-token\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.765817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.765774 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-kube-api-access-vq7qh\") pod \"e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:35.950707 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:35.950601 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:36.090846 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.090817 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx"] Jul 15 13:41:36.092652 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:41:36.092623 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ab63831_3790_43f8_8203_19424aa030d4.slice/crio-08cbde478a640a733182be24ef0bf477a8f9f34888fc685d9a530d492b866eac WatchSource:0}: Error finding container 08cbde478a640a733182be24ef0bf477a8f9f34888fc685d9a530d492b866eac: Status 404 returned error can't find the container with id 08cbde478a640a733182be24ef0bf477a8f9f34888fc685d9a530d492b866eac Jul 15 13:41:36.094919 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.094885 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:41:36.095025 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.094951 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:41:36.095025 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.094980 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:41:36.721091 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.721054 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" event={"ID":"2ab63831-3790-43f8-8203-19424aa030d4","Type":"ContainerStarted","Data":"ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd"} Jul 15 13:41:36.721262 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.721106 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" event={"ID":"2ab63831-3790-43f8-8203-19424aa030d4","Type":"ContainerStarted","Data":"08cbde478a640a733182be24ef0bf477a8f9f34888fc685d9a530d492b866eac"} Jul 15 13:41:36.748346 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.748295 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" podStartSLOduration=1.748281538 podStartE2EDuration="1.748281538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:41:36.745652766 +0000 UTC m=+2102.701107355" watchObservedRunningTime="2026-07-15 13:41:36.748281538 +0000 UTC m=+2102.703736105" Jul 15 13:41:36.951200 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:36.951166 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:37.956805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:37.956778 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:38.737699 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:38.733062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:38.738794 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:38.738766 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:50.787352 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.787320 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq"] Jul 15 13:41:50.787837 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.787809 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" Jul 15 13:41:50.787837 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.787832 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="72a90e41-5926-4c0a-b101-0a9aeee8c9d3" containerName="istio-proxy" Jul 15 13:41:50.791536 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.791518 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:50.794563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.794527 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 15 13:41:50.794777 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.794752 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:41:50.802249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.802138 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq"] Jul 15 13:41:50.908853 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:50.908823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62852d60-3940-4069-a66c-dde10bd6d94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq\" (UID: \"62852d60-3940-4069-a66c-dde10bd6d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:51.010201 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.010166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62852d60-3940-4069-a66c-dde10bd6d94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq\" (UID: \"62852d60-3940-4069-a66c-dde10bd6d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:51.010925 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.010898 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62852d60-3940-4069-a66c-dde10bd6d94b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq\" (UID: \"62852d60-3940-4069-a66c-dde10bd6d94b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:51.109685 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.109612 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:51.244447 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.244236 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq"] Jul 15 13:41:51.246553 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:41:51.246524 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62852d60_3940_4069_a66c_dde10bd6d94b.slice/crio-3128ecdc8b1a705676a003e000c905143544c040423524aebc41fed3f49cea7b WatchSource:0}: Error finding container 3128ecdc8b1a705676a003e000c905143544c040423524aebc41fed3f49cea7b: Status 404 returned error can't find the container with id 3128ecdc8b1a705676a003e000c905143544c040423524aebc41fed3f49cea7b Jul 15 13:41:51.805181 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.805149 2564 generic.go:356] "Generic (PLEG): container finished" podID="2ab63831-3790-43f8-8203-19424aa030d4" containerID="ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd" exitCode=0 Jul 15 13:41:51.805644 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.805228 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" event={"ID":"2ab63831-3790-43f8-8203-19424aa030d4","Type":"ContainerDied","Data":"ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd"} Jul 15 13:41:51.805737 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.805721 2564 scope.go:117] "RemoveContainer" containerID="ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd" Jul 15 13:41:51.806747 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.806722 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:41:51.806850 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.806791 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:41:51.806850 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.806823 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:41:51.807194 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.807172 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" event={"ID":"62852d60-3940-4069-a66c-dde10bd6d94b","Type":"ContainerStarted","Data":"dba3c3a7c547f791da6ff80b279e1a54f7cc6ce8e6b538ae540998a12756752c"} Jul 15 13:41:51.807295 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.807230 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" event={"ID":"62852d60-3940-4069-a66c-dde10bd6d94b","Type":"ContainerStarted","Data":"3128ecdc8b1a705676a003e000c905143544c040423524aebc41fed3f49cea7b"} Jul 15 13:41:51.841579 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:51.841516 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" podStartSLOduration=1.8414977750000001 podStartE2EDuration="1.841497775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:41:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:41:51.839628787 +0000 UTC m=+2117.795083353" watchObservedRunningTime="2026-07-15 13:41:51.841497775 +0000 UTC m=+2117.796952339" Jul 15 13:41:52.813816 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:52.813777 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" event={"ID":"2ab63831-3790-43f8-8203-19424aa030d4","Type":"ContainerStarted","Data":"3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57"} Jul 15 13:41:52.815273 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:52.815251 2564 generic.go:356] "Generic (PLEG): container finished" podID="62852d60-3940-4069-a66c-dde10bd6d94b" containerID="dba3c3a7c547f791da6ff80b279e1a54f7cc6ce8e6b538ae540998a12756752c" exitCode=0 Jul 15 13:41:52.815394 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:52.815338 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" event={"ID":"62852d60-3940-4069-a66c-dde10bd6d94b","Type":"ContainerDied","Data":"dba3c3a7c547f791da6ff80b279e1a54f7cc6ce8e6b538ae540998a12756752c"} Jul 15 13:41:52.951719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:52.951689 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:52.952152 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:52.952125 2564 patch_prober.go:28] interesting pod/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 13:41:52.952280 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:52.952177 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 13:41:53.733893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:53.733849 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:53.956504 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:53.953844 2564 patch_prober.go:28] interesting pod/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 13:41:53.956504 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:53.953907 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 13:41:53.984772 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:53.984714 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:54.049831 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.049790 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62852d60-3940-4069-a66c-dde10bd6d94b-maas-api-service-ca\") pod \"62852d60-3940-4069-a66c-dde10bd6d94b\" (UID: \"62852d60-3940-4069-a66c-dde10bd6d94b\") " Jul 15 13:41:54.050824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.050796 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62852d60-3940-4069-a66c-dde10bd6d94b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "62852d60-3940-4069-a66c-dde10bd6d94b" (UID: "62852d60-3940-4069-a66c-dde10bd6d94b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:41:54.152011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.151966 2564 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62852d60-3940-4069-a66c-dde10bd6d94b-maas-api-service-ca\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:41:54.827282 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.827252 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" Jul 15 13:41:54.827427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.827287 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq" event={"ID":"62852d60-3940-4069-a66c-dde10bd6d94b","Type":"ContainerDied","Data":"3128ecdc8b1a705676a003e000c905143544c040423524aebc41fed3f49cea7b"} Jul 15 13:41:54.827427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.827334 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3128ecdc8b1a705676a003e000c905143544c040423524aebc41fed3f49cea7b" Jul 15 13:41:54.951880 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.951798 2564 patch_prober.go:28] interesting pod/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 13:41:54.952115 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:54.952086 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 13:41:55.951668 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:55.951629 2564 patch_prober.go:28] interesting pod/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body= Jul 15 13:41:55.952238 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:55.952207 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" Jul 15 13:41:56.955087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:56.955055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:41:56.956051 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:41:56.956033 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:42:16.552479 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:16.552425 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx"] Jul 15 13:42:16.553080 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:16.552654 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" containerID="cri-o://3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57" gracePeriod=30 Jul 15 13:42:21.809911 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.809879 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:42:21.918745 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918648 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-envoy\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.918745 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918718 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab63831-3790-43f8-8203-19424aa030d4-istio-podinfo\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.918745 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918743 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-socket\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919045 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918782 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab63831-3790-43f8-8203-19424aa030d4-istiod-ca-cert\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919045 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918821 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-credential-socket\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919045 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918871 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-kube-api-access-vq7qh\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919045 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.918915 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-data\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919080 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-istio-token\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919093 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:42:21.919249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919171 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-certs\") pod \"2ab63831-3790-43f8-8203-19424aa030d4\" (UID: \"2ab63831-3790-43f8-8203-19424aa030d4\") " Jul 15 13:42:21.919249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919185 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:42:21.919249 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919222 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ab63831-3790-43f8-8203-19424aa030d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:42:21.919537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919518 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:21.919598 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919542 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2ab63831-3790-43f8-8203-19424aa030d4-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:21.919598 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919556 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:21.919598 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919572 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:42:21.920011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.919985 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:42:21.920964 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.920940 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:42:21.921209 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.921186 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-kube-api-access-vq7qh" (OuterVolumeSpecName: "kube-api-access-vq7qh") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "kube-api-access-vq7qh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:42:21.921439 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.921422 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:42:21.921499 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.921438 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2ab63831-3790-43f8-8203-19424aa030d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "2ab63831-3790-43f8-8203-19424aa030d4" (UID: "2ab63831-3790-43f8-8203-19424aa030d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:42:21.986135 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.986103 2564 generic.go:356] "Generic (PLEG): container finished" podID="2ab63831-3790-43f8-8203-19424aa030d4" containerID="3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57" exitCode=0 Jul 15 13:42:21.986297 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.986150 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" Jul 15 13:42:21.986297 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.986165 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" event={"ID":"2ab63831-3790-43f8-8203-19424aa030d4","Type":"ContainerDied","Data":"3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57"} Jul 15 13:42:21.986297 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.986208 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx" event={"ID":"2ab63831-3790-43f8-8203-19424aa030d4","Type":"ContainerDied","Data":"08cbde478a640a733182be24ef0bf477a8f9f34888fc685d9a530d492b866eac"} Jul 15 13:42:21.986297 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.986231 2564 scope.go:117] "RemoveContainer" containerID="3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57" Jul 15 13:42:21.995736 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:21.995718 2564 scope.go:117] "RemoveContainer" containerID="ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd" Jul 15 13:42:22.003954 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.003934 2564 scope.go:117] "RemoveContainer" containerID="3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57" Jul 15 13:42:22.004248 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:42:22.004226 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57\": container with ID starting with 3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57 not found: ID does not exist" containerID="3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57" Jul 15 13:42:22.004493 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.004256 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57"} err="failed to get container status \"3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57\": rpc error: code = NotFound desc = could not find container \"3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57\": container with ID starting with 3944bfc5f2b0dd636ab6d4fd7c0dd74a5386762b0b9708e9ce27a491d09c1d57 not found: ID does not exist" Jul 15 13:42:22.004493 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.004283 2564 scope.go:117] "RemoveContainer" containerID="ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd" Jul 15 13:42:22.004673 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:42:22.004611 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd\": container with ID starting with ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd not found: ID does not exist" containerID="ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd" Jul 15 13:42:22.004673 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.004641 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd"} err="failed to get container status \"ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd\": rpc error: code = NotFound desc = could not find container \"ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd\": container with ID starting with ca5018c8871c1fff7d3559d327239e9e5f99116cc9c7528637a09725b53a00fd not found: ID does not exist" Jul 15 13:42:22.007097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.007075 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx"] Jul 15 13:42:22.010748 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.010730 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b14f8bd8-openshift-default-54fdd78547-lkqnx"] Jul 15 13:42:22.020850 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.020831 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq7qh\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-kube-api-access-vq7qh\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:22.020850 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.020850 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:22.020967 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.020859 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2ab63831-3790-43f8-8203-19424aa030d4-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:22.020967 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.020869 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:22.020967 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.020877 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2ab63831-3790-43f8-8203-19424aa030d4-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:22.020967 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.020885 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2ab63831-3790-43f8-8203-19424aa030d4-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:42:22.658424 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:22.658385 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ab63831-3790-43f8-8203-19424aa030d4" path="/var/lib/kubelet/pods/2ab63831-3790-43f8-8203-19424aa030d4/volumes" Jul 15 13:42:34.749965 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:34.749890 2564 scope.go:117] "RemoveContainer" containerID="df78d5b4f2d2ae3cab4ac12b8eabc0edf24946e14a92011039b07d1c96c638e7" Jul 15 13:42:36.505844 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.505810 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf"] Jul 15 13:42:36.506638 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506611 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62852d60-3940-4069-a66c-dde10bd6d94b" containerName="revoke-keys" Jul 15 13:42:36.506638 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506638 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="62852d60-3940-4069-a66c-dde10bd6d94b" containerName="revoke-keys" Jul 15 13:42:36.506877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506660 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" Jul 15 13:42:36.506877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506670 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" Jul 15 13:42:36.506877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506790 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" Jul 15 13:42:36.506877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506801 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" Jul 15 13:42:36.506877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.506812 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="62852d60-3940-4069-a66c-dde10bd6d94b" containerName="revoke-keys" Jul 15 13:42:36.520498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.520444 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.522929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.522890 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf"] Jul 15 13:42:36.523505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.523451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e86098b2-openshift-default-dockercfg-mphv5\"" Jul 15 13:42:36.523884 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.523860 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:42:36.663459 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663427 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-envoy\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-token\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663537 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-socket\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjzkr\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-kube-api-access-mjzkr\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663599 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-certs\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663619 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3c6d3d-2622-4604-8af5-869ca3d55240-istiod-ca-cert\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663649 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663645 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-podinfo\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663847 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663682 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-credential-socket\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.663847 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.663705 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-data\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.764643 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.764554 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-credential-socket\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.764643 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.764602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-data\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.764834 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.764725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-envoy\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.764834 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.764761 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-token\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.764945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.764908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-socket\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.764976 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjzkr\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-kube-api-access-mjzkr\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765019 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-certs\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765037 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-credential-socket\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765206 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765054 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3c6d3d-2622-4604-8af5-869ca3d55240-istiod-ca-cert\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765206 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765087 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-podinfo\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765206 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765107 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-data\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765730 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765323 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-socket\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765730 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765344 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-certs\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.765898 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.765873 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3c6d3d-2622-4604-8af5-869ca3d55240-istiod-ca-cert\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.767127 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.767104 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-envoy\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.767583 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.767565 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-podinfo\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.772817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.772793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-token\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.773160 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.773144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjzkr\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-kube-api-access-mjzkr\") pod \"e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.832984 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.832950 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:36.974820 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.974789 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf"] Jul 15 13:42:36.976570 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:42:36.976539 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a3c6d3d_2622_4604_8af5_869ca3d55240.slice/crio-32c943990bb19e10861b307c7582a5d2e6d4c40345fc29980ce3eb2488e4dfd0 WatchSource:0}: Error finding container 32c943990bb19e10861b307c7582a5d2e6d4c40345fc29980ce3eb2488e4dfd0: Status 404 returned error can't find the container with id 32c943990bb19e10861b307c7582a5d2e6d4c40345fc29980ce3eb2488e4dfd0 Jul 15 13:42:36.978737 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.978707 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:42:36.978823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.978772 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:42:36.978823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:36.978799 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:42:37.076106 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:37.076062 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" event={"ID":"9a3c6d3d-2622-4604-8af5-869ca3d55240","Type":"ContainerStarted","Data":"acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013"} Jul 15 13:42:37.076282 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:37.076139 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" event={"ID":"9a3c6d3d-2622-4604-8af5-869ca3d55240","Type":"ContainerStarted","Data":"32c943990bb19e10861b307c7582a5d2e6d4c40345fc29980ce3eb2488e4dfd0"} Jul 15 13:42:37.097410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:37.097363 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podStartSLOduration=1.097345631 podStartE2EDuration="1.097345631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:42:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:42:37.096098449 +0000 UTC m=+2163.051553049" watchObservedRunningTime="2026-07-15 13:42:37.097345631 +0000 UTC m=+2163.052800194" Jul 15 13:42:37.833088 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:37.833057 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:38.839122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:38.839094 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:39.087750 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:39.087715 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:39.088800 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:39.088777 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:51.084570 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.084533 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj"] Jul 15 13:42:51.085290 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.085270 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" Jul 15 13:42:51.085290 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.085296 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ab63831-3790-43f8-8203-19424aa030d4" containerName="istio-proxy" Jul 15 13:42:51.088703 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.088683 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.091241 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.091223 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e86098b2\"/\"kube-root-ca.crt\"" Jul 15 13:42:51.092162 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.092144 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e86098b2\"/\"default-dockercfg-g9w4t\"" Jul 15 13:42:51.092244 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.092219 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-e86098b2\"/\"openshift-service-ca.crt\"" Jul 15 13:42:51.092244 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.092228 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-e86098b2\"/\"e2e-lifecycle-model-e86098b2-kserve-self-signed-certs\"" Jul 15 13:42:51.098885 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.098864 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj"] Jul 15 13:42:51.215136 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.215105 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-dshm\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.215257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.215148 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-model-cache\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.215257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.215196 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgd2n\" (UniqueName: \"kubernetes.io/projected/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-kube-api-access-vgd2n\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.215346 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.215276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tls-certs\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.215395 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.215357 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tmp-dir\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.215497 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.215465 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-home\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316397 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tmp-dir\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-home\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-dshm\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316486 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-model-cache\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316592 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316510 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgd2n\" (UniqueName: \"kubernetes.io/projected/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-kube-api-access-vgd2n\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316640 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tls-certs\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316915 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316894 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-home\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.316977 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316943 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tmp-dir\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.317014 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.316976 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-model-cache\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.318709 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.318683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-dshm\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.319137 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.319121 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tls-certs\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.323776 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.323752 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgd2n\" (UniqueName: \"kubernetes.io/projected/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-kube-api-access-vgd2n\") pod \"e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.399706 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.399614 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:42:51.550620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:51.550591 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj"] Jul 15 13:42:51.552444 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:42:51.552414 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17d75c71_838f_4ba6_a9dc_f96a08d7a66b.slice/crio-d8380b6d1f85ba89db6224f2852238427acd402ea831c2aa6b626a2df7207ce8 WatchSource:0}: Error finding container d8380b6d1f85ba89db6224f2852238427acd402ea831c2aa6b626a2df7207ce8: Status 404 returned error can't find the container with id d8380b6d1f85ba89db6224f2852238427acd402ea831c2aa6b626a2df7207ce8 Jul 15 13:42:52.167252 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:52.167213 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" event={"ID":"17d75c71-838f-4ba6-a9dc-f96a08d7a66b","Type":"ContainerStarted","Data":"bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353"} Jul 15 13:42:52.167687 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:52.167260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" event={"ID":"17d75c71-838f-4ba6-a9dc-f96a08d7a66b","Type":"ContainerStarted","Data":"d8380b6d1f85ba89db6224f2852238427acd402ea831c2aa6b626a2df7207ce8"} Jul 15 13:42:52.187524 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:52.187443 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" podStartSLOduration=1.187425033 podStartE2EDuration="1.187425033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:42:52.184314784 +0000 UTC m=+2178.139769352" watchObservedRunningTime="2026-07-15 13:42:52.187425033 +0000 UTC m=+2178.142879597" Jul 15 13:42:54.089528 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.089418 2564 patch_prober.go:28] interesting pod/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 13:42:54.089528 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.089484 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 13:42:54.179940 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.179909 2564 generic.go:356] "Generic (PLEG): container finished" podID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerID="acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013" exitCode=0 Jul 15 13:42:54.180105 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.179988 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" event={"ID":"9a3c6d3d-2622-4604-8af5-869ca3d55240","Type":"ContainerDied","Data":"acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013"} Jul 15 13:42:54.180394 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.180379 2564 scope.go:117] "RemoveContainer" containerID="acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013" Jul 15 13:42:54.181504 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.181457 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:42:54.181575 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.181557 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:42:54.181623 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:54.181601 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:42:55.186890 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:55.186855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" event={"ID":"9a3c6d3d-2622-4604-8af5-869ca3d55240","Type":"ContainerStarted","Data":"0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f"} Jul 15 13:42:55.833987 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:55.833945 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:55.834262 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:55.834224 2564 patch_prober.go:28] interesting pod/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 13:42:55.834358 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:55.834294 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 13:42:56.834392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:56.834350 2564 patch_prober.go:28] interesting pod/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 13:42:56.835035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:56.835000 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 13:42:57.833646 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:57.833602 2564 patch_prober.go:28] interesting pod/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 13:42:57.833863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:57.833669 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 13:42:58.834368 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:58.834328 2564 patch_prober.go:28] interesting pod/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" start-of-body= Jul 15 13:42:58.835003 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:58.834968 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.52:15021/healthz/ready\": dial tcp 10.134.0.52:15021: connect: connection refused" Jul 15 13:42:59.837618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:59.837589 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:59.837992 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:59.837892 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:42:59.838597 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:42:59.838580 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:43:01.400661 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:01.400625 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:43:01.401054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:01.400697 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:43:01.413440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:01.413411 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:43:02.236844 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:02.236805 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:43:34.773950 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:34.773923 2564 scope.go:117] "RemoveContainer" containerID="c41e67c3964c12ac781269972d38a5a0996f0409a2786710e2bdb8cc4177c1e9" Jul 15 13:43:41.886082 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:41.886043 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj"] Jul 15 13:43:56.906553 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:56.906458 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" podUID="17d75c71-838f-4ba6-a9dc-f96a08d7a66b" containerName="main" containerID="cri-o://bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353" gracePeriod=45 Jul 15 13:43:57.156066 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.156043 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:43:57.237056 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.236981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vgd2n\" (UniqueName: \"kubernetes.io/projected/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-kube-api-access-vgd2n\") pod \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " Jul 15 13:43:57.237215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237065 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-dshm\") pod \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " Jul 15 13:43:57.237215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237113 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-model-cache\") pod \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " Jul 15 13:43:57.237215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237145 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tls-certs\") pod \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " Jul 15 13:43:57.237215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237176 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tmp-dir\") pod \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " Jul 15 13:43:57.237215 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237200 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-home\") pod \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\" (UID: \"17d75c71-838f-4ba6-a9dc-f96a08d7a66b\") " Jul 15 13:43:57.237514 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237383 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-model-cache" (OuterVolumeSpecName: "model-cache") pod "17d75c71-838f-4ba6-a9dc-f96a08d7a66b" (UID: "17d75c71-838f-4ba6-a9dc-f96a08d7a66b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:43:57.237584 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237530 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "17d75c71-838f-4ba6-a9dc-f96a08d7a66b" (UID: "17d75c71-838f-4ba6-a9dc-f96a08d7a66b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:43:57.237584 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237545 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-home" (OuterVolumeSpecName: "home") pod "17d75c71-838f-4ba6-a9dc-f96a08d7a66b" (UID: "17d75c71-838f-4ba6-a9dc-f96a08d7a66b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:43:57.237682 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237662 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-model-cache\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:43:57.237719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237682 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tmp-dir\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:43:57.237719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.237694 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-home\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:43:57.239159 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.239140 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-dshm" (OuterVolumeSpecName: "dshm") pod "17d75c71-838f-4ba6-a9dc-f96a08d7a66b" (UID: "17d75c71-838f-4ba6-a9dc-f96a08d7a66b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:43:57.239234 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.239181 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "17d75c71-838f-4ba6-a9dc-f96a08d7a66b" (UID: "17d75c71-838f-4ba6-a9dc-f96a08d7a66b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:43:57.239285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.239267 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-kube-api-access-vgd2n" (OuterVolumeSpecName: "kube-api-access-vgd2n") pod "17d75c71-838f-4ba6-a9dc-f96a08d7a66b" (UID: "17d75c71-838f-4ba6-a9dc-f96a08d7a66b"). InnerVolumeSpecName "kube-api-access-vgd2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:43:57.338810 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.338769 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-dshm\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:43:57.338810 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.338805 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-tls-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:43:57.338810 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.338815 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgd2n\" (UniqueName: \"kubernetes.io/projected/17d75c71-838f-4ba6-a9dc-f96a08d7a66b-kube-api-access-vgd2n\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:43:57.520820 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.520746 2564 generic.go:356] "Generic (PLEG): container finished" podID="17d75c71-838f-4ba6-a9dc-f96a08d7a66b" containerID="bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353" exitCode=0 Jul 15 13:43:57.520820 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.520792 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" Jul 15 13:43:57.520820 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.520812 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" event={"ID":"17d75c71-838f-4ba6-a9dc-f96a08d7a66b","Type":"ContainerDied","Data":"bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353"} Jul 15 13:43:57.521010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.520850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj" event={"ID":"17d75c71-838f-4ba6-a9dc-f96a08d7a66b","Type":"ContainerDied","Data":"d8380b6d1f85ba89db6224f2852238427acd402ea831c2aa6b626a2df7207ce8"} Jul 15 13:43:57.521010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.520873 2564 scope.go:117] "RemoveContainer" containerID="bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353" Jul 15 13:43:57.529809 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.529789 2564 scope.go:117] "RemoveContainer" containerID="bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353" Jul 15 13:43:57.530071 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:43:57.530053 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353\": container with ID starting with bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353 not found: ID does not exist" containerID="bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353" Jul 15 13:43:57.530146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.530088 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353"} err="failed to get container status \"bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353\": rpc error: code = NotFound desc = could not find container \"bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353\": container with ID starting with bbe3d00dc78ad579df8daa1ff39f815cb7a5acd0a3573d298dba8401ac18c353 not found: ID does not exist" Jul 15 13:43:57.541664 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.541644 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj"] Jul 15 13:43:57.547183 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:57.547154 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-e86098b2/e2e-lifecycle-model-e86098b2-kserve-779476f459-qj8rj"] Jul 15 13:43:58.657703 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:43:58.657667 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17d75c71-838f-4ba6-a9dc-f96a08d7a66b" path="/var/lib/kubelet/pods/17d75c71-838f-4ba6-a9dc-f96a08d7a66b/volumes" Jul 15 13:44:29.164885 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:29.164855 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf"] Jul 15 13:44:29.165382 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:29.165112 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" containerID="cri-o://0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f" gracePeriod=30 Jul 15 13:44:29.840379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:29.840347 2564 patch_prober.go:28] interesting pod/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 13:44:29.840551 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:29.840400 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:44:34.426843 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.426786 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:44:34.575996 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.575946 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-podinfo\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.575996 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.575996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-data\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576255 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576038 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-socket\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576255 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576144 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-envoy\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576255 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576192 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjzkr\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-kube-api-access-mjzkr\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576255 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576222 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3c6d3d-2622-4604-8af5-869ca3d55240-istiod-ca-cert\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576501 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576260 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-credential-socket\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576501 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576304 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-token\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.576501 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576326 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-certs\") pod \"9a3c6d3d-2622-4604-8af5-869ca3d55240\" (UID: \"9a3c6d3d-2622-4604-8af5-869ca3d55240\") " Jul 15 13:44:34.577109 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.576871 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:44:34.577109 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.577021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:44:34.577109 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.577036 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a3c6d3d-2622-4604-8af5-869ca3d55240-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:44:34.579835 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.579795 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:44:34.580795 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.580769 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-data" (OuterVolumeSpecName: "istio-data") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:44:34.582089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.581723 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-token" (OuterVolumeSpecName: "istio-token") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:44:34.582429 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.582408 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:44:34.582549 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.582514 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:44:34.588893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.588728 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-kube-api-access-mjzkr" (OuterVolumeSpecName: "kube-api-access-mjzkr") pod "9a3c6d3d-2622-4604-8af5-869ca3d55240" (UID: "9a3c6d3d-2622-4604-8af5-869ca3d55240"). InnerVolumeSpecName "kube-api-access-mjzkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677305 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677330 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjzkr\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-kube-api-access-mjzkr\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677340 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9a3c6d3d-2622-4604-8af5-869ca3d55240-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677349 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677358 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677366 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677371 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677373 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677861 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677381 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.677861 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.677390 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9a3c6d3d-2622-4604-8af5-869ca3d55240-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:44:34.718709 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.718672 2564 generic.go:356] "Generic (PLEG): container finished" podID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerID="0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f" exitCode=0 Jul 15 13:44:34.718877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.718721 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" Jul 15 13:44:34.718877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.718763 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" event={"ID":"9a3c6d3d-2622-4604-8af5-869ca3d55240","Type":"ContainerDied","Data":"0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f"} Jul 15 13:44:34.718877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.718821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf" event={"ID":"9a3c6d3d-2622-4604-8af5-869ca3d55240","Type":"ContainerDied","Data":"32c943990bb19e10861b307c7582a5d2e6d4c40345fc29980ce3eb2488e4dfd0"} Jul 15 13:44:34.718877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.718845 2564 scope.go:117] "RemoveContainer" containerID="0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f" Jul 15 13:44:34.730388 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.730365 2564 scope.go:117] "RemoveContainer" containerID="acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013" Jul 15 13:44:34.736281 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.736257 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf"] Jul 15 13:44:34.740550 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.740530 2564 scope.go:117] "RemoveContainer" containerID="0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f" Jul 15 13:44:34.740869 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:44:34.740842 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f\": container with ID starting with 0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f not found: ID does not exist" containerID="0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f" Jul 15 13:44:34.740991 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.740873 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f"} err="failed to get container status \"0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f\": rpc error: code = NotFound desc = could not find container \"0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f\": container with ID starting with 0b0230e006befe44c65a7b69229c7e16037c2be8b238f0410a86886f87a8319f not found: ID does not exist" Jul 15 13:44:34.740991 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.740891 2564 scope.go:117] "RemoveContainer" containerID="acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013" Jul 15 13:44:34.741129 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:44:34.741117 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013\": container with ID starting with acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013 not found: ID does not exist" containerID="acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013" Jul 15 13:44:34.741189 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.741145 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013"} err="failed to get container status \"acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013\": rpc error: code = NotFound desc = could not find container \"acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013\": container with ID starting with acfe3619565a8090619ff0296ca41e141e5a91e8a9ef12bfcd623ced0fe76013 not found: ID does not exist" Jul 15 13:44:34.741545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:34.741525 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e86098b2-openshift-default-6c74c9895-nxshf"] Jul 15 13:44:36.657705 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:44:36.657673 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" path="/var/lib/kubelet/pods/9a3c6d3d-2622-4604-8af5-869ca3d55240/volumes" Jul 15 13:45:00.143895 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.143862 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4"] Jul 15 13:45:00.144310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144258 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" Jul 15 13:45:00.144310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144268 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" Jul 15 13:45:00.144310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144285 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" Jul 15 13:45:00.144310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144291 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" Jul 15 13:45:00.144310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144302 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17d75c71-838f-4ba6-a9dc-f96a08d7a66b" containerName="main" Jul 15 13:45:00.144310 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144307 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="17d75c71-838f-4ba6-a9dc-f96a08d7a66b" containerName="main" Jul 15 13:45:00.144525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144371 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" Jul 15 13:45:00.144525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.144381 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="17d75c71-838f-4ba6-a9dc-f96a08d7a66b" containerName="main" Jul 15 13:45:00.146456 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.146441 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" Jul 15 13:45:00.148789 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.148764 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-f6nkx\"" Jul 15 13:45:00.154442 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.154423 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4"] Jul 15 13:45:00.157930 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.157911 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" Jul 15 13:45:00.291380 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.291356 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4"] Jul 15 13:45:00.293722 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:45:00.293694 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83db9a48_aee8_4d05_8a0a_094740b6b5ac.slice/crio-9b302f913fbb89c4313ad28f01b07c5943217203800da859935716c44feb77b6 WatchSource:0}: Error finding container 9b302f913fbb89c4313ad28f01b07c5943217203800da859935716c44feb77b6: Status 404 returned error can't find the container with id 9b302f913fbb89c4313ad28f01b07c5943217203800da859935716c44feb77b6 Jul 15 13:45:00.851677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.851591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" event={"ID":"83db9a48-aee8-4d05-8a0a-094740b6b5ac","Type":"ContainerStarted","Data":"7797d0a6ff074787fa53268870eb0fc8206d42e6b5d121e8f7be0d1a0f4ff0ef"} Jul 15 13:45:00.851677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.851647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" event={"ID":"83db9a48-aee8-4d05-8a0a-094740b6b5ac","Type":"ContainerStarted","Data":"9b302f913fbb89c4313ad28f01b07c5943217203800da859935716c44feb77b6"} Jul 15 13:45:00.872544 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:00.872461 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" podStartSLOduration=0.872442123 podStartE2EDuration="872.442123ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:45:00.867463684 +0000 UTC m=+2306.822918248" watchObservedRunningTime="2026-07-15 13:45:00.872442123 +0000 UTC m=+2306.827896687" Jul 15 13:45:01.858111 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:01.858057 2564 generic.go:356] "Generic (PLEG): container finished" podID="83db9a48-aee8-4d05-8a0a-094740b6b5ac" containerID="7797d0a6ff074787fa53268870eb0fc8206d42e6b5d121e8f7be0d1a0f4ff0ef" exitCode=0 Jul 15 13:45:01.858542 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:01.858133 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" event={"ID":"83db9a48-aee8-4d05-8a0a-094740b6b5ac","Type":"ContainerDied","Data":"7797d0a6ff074787fa53268870eb0fc8206d42e6b5d121e8f7be0d1a0f4ff0ef"} Jul 15 13:45:02.996910 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:02.996886 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" Jul 15 13:45:03.869439 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:03.869411 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" Jul 15 13:45:03.869673 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:03.869457 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29735385-k2wb4" event={"ID":"83db9a48-aee8-4d05-8a0a-094740b6b5ac","Type":"ContainerDied","Data":"9b302f913fbb89c4313ad28f01b07c5943217203800da859935716c44feb77b6"} Jul 15 13:45:03.869673 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:03.869532 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b302f913fbb89c4313ad28f01b07c5943217203800da859935716c44feb77b6" Jul 15 13:45:24.683188 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.683161 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28"] Jul 15 13:45:24.683636 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.683558 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83db9a48-aee8-4d05-8a0a-094740b6b5ac" containerName="cleanup" Jul 15 13:45:24.683636 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.683569 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="83db9a48-aee8-4d05-8a0a-094740b6b5ac" containerName="cleanup" Jul 15 13:45:24.683748 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.683643 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a3c6d3d-2622-4604-8af5-869ca3d55240" containerName="istio-proxy" Jul 15 13:45:24.683748 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.683665 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="83db9a48-aee8-4d05-8a0a-094740b6b5ac" containerName="cleanup" Jul 15 13:45:24.686989 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.686970 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.689227 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.689206 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-486f14-openshift-default-dockercfg-dz26q\"" Jul 15 13:45:24.689345 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.689210 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 15 13:45:24.701181 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.701153 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28"] Jul 15 13:45:24.844708 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844677 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b6046b-b229-44dc-9694-4b9c920504f4-istiod-ca-cert\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.844893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844738 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-data\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.844893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-socket\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.844893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-token\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.844893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzk7w\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-kube-api-access-xzk7w\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.844893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844883 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-credential-socket\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.845093 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-certs\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.845093 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.844958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-podinfo\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.845093 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.845019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-envoy\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.945893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945812 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-data\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.945893 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945847 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-socket\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946095 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-token\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946095 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945921 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzk7w\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-kube-api-access-xzk7w\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946095 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945945 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-credential-socket\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946095 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-certs\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946095 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.945998 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-podinfo\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946095 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-envoy\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946126 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b6046b-b229-44dc-9694-4b9c920504f4-istiod-ca-cert\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-data\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-socket\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946389 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-credential-socket\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946648 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946498 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-certs\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.946920 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.946899 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b6046b-b229-44dc-9694-4b9c920504f4-istiod-ca-cert\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.948431 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.948413 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-envoy\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.948855 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.948840 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-podinfo\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.958662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.958639 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-token\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.958784 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.958718 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzk7w\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-kube-api-access-xzk7w\") pod \"e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:24.998617 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:24.998592 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:25.137277 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.137244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28"] Jul 15 13:45:25.138257 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:45:25.138229 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b6046b_b229_44dc_9694_4b9c920504f4.slice/crio-1635eac0a19de4aa397a3633b1530af314cdda25b000f9f33d922ea768a7da72 WatchSource:0}: Error finding container 1635eac0a19de4aa397a3633b1530af314cdda25b000f9f33d922ea768a7da72: Status 404 returned error can't find the container with id 1635eac0a19de4aa397a3633b1530af314cdda25b000f9f33d922ea768a7da72 Jul 15 13:45:25.140597 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.140561 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:25.140695 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.140679 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:25.140739 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.140724 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:25.989432 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.989405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" event={"ID":"c1b6046b-b229-44dc-9694-4b9c920504f4","Type":"ContainerStarted","Data":"53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58"} Jul 15 13:45:25.989801 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.989452 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" event={"ID":"c1b6046b-b229-44dc-9694-4b9c920504f4","Type":"ContainerStarted","Data":"1635eac0a19de4aa397a3633b1530af314cdda25b000f9f33d922ea768a7da72"} Jul 15 13:45:25.999257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:25.999238 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:26.009622 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:26.009574 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podStartSLOduration=2.009555733 podStartE2EDuration="2.009555733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:45:26.007946141 +0000 UTC m=+2331.963400704" watchObservedRunningTime="2026-07-15 13:45:26.009555733 +0000 UTC m=+2331.965010298" Jul 15 13:45:27.004631 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:27.004599 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:27.999390 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:27.999360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:28.000294 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:28.000273 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:33.063713 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.063683 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq"] Jul 15 13:45:33.069175 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.069152 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.071521 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.071498 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-a5d1bc-openshift-default-dockercfg-kgjs6\"" Jul 15 13:45:33.080840 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.080814 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq"] Jul 15 13:45:33.124061 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124038 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaee3805-1407-4fff-bfc5-c821aef0eecd-istiod-ca-cert\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124180 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124071 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-socket\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124180 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-envoy\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124180 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124128 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-podinfo\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124179 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-credential-socket\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124214 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-certs\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124246 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gd7q\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-kube-api-access-6gd7q\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124285 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124270 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-data\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.124410 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.124312 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-token\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225224 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225190 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gd7q\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-kube-api-access-6gd7q\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225224 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225227 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-data\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-token\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225273 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaee3805-1407-4fff-bfc5-c821aef0eecd-istiod-ca-cert\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225308 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-socket\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225338 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-envoy\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225421 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225388 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-podinfo\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225704 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225448 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-credential-socket\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225704 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-certs\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225813 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225775 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-socket\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225813 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-certs\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225913 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-credential-socket\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.225913 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-data\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.226017 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.225998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaee3805-1407-4fff-bfc5-c821aef0eecd-istiod-ca-cert\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.227713 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.227692 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-envoy\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.227848 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.227826 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-podinfo\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.232844 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.232820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gd7q\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-kube-api-access-6gd7q\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.233012 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.232998 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-token\") pod \"e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.382640 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.382573 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:33.519712 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.519683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq"] Jul 15 13:45:33.521332 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:45:33.521303 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaee3805_1407_4fff_bfc5_c821aef0eecd.slice/crio-4951f87f7d5cb749e2000d1bf28b4059949ede8e9ac8dfe4e7322fe693c7e8e1 WatchSource:0}: Error finding container 4951f87f7d5cb749e2000d1bf28b4059949ede8e9ac8dfe4e7322fe693c7e8e1: Status 404 returned error can't find the container with id 4951f87f7d5cb749e2000d1bf28b4059949ede8e9ac8dfe4e7322fe693c7e8e1 Jul 15 13:45:33.523456 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.523427 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:33.523526 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.523502 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:33.523569 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:33.523533 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:34.031890 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:34.031857 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" event={"ID":"eaee3805-1407-4fff-bfc5-c821aef0eecd","Type":"ContainerStarted","Data":"9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7"} Jul 15 13:45:34.032056 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:34.031912 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" event={"ID":"eaee3805-1407-4fff-bfc5-c821aef0eecd","Type":"ContainerStarted","Data":"4951f87f7d5cb749e2000d1bf28b4059949ede8e9ac8dfe4e7322fe693c7e8e1"} Jul 15 13:45:34.056492 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:34.056422 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podStartSLOduration=1.056402934 podStartE2EDuration="1.056402934s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:45:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:45:34.053240213 +0000 UTC m=+2340.008694808" watchObservedRunningTime="2026-07-15 13:45:34.056402934 +0000 UTC m=+2340.011857499" Jul 15 13:45:34.383538 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:34.383506 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:34.388183 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:34.388158 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:34.813722 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:34.813695 2564 scope.go:117] "RemoveContainer" containerID="6ba158b830ddd6dd0802c1067d60bc19a4aa5e5cd2b55b8ed8558387af81d1bb" Jul 15 13:45:35.040016 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:35.039990 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:35.040982 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:35.040962 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:43.083773 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:43.083741 2564 patch_prober.go:28] interesting pod/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:42364->10.134.0.55:15021: read: connection reset by peer" start-of-body= Jul 15 13:45:43.084211 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:43.083817 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": read tcp 10.134.0.2:42364->10.134.0.55:15021: read: connection reset by peer" Jul 15 13:45:44.094509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:44.094463 2564 generic.go:356] "Generic (PLEG): container finished" podID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerID="53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58" exitCode=0 Jul 15 13:45:44.094509 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:44.094504 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" event={"ID":"c1b6046b-b229-44dc-9694-4b9c920504f4","Type":"ContainerDied","Data":"53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58"} Jul 15 13:45:44.094966 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:44.094926 2564 scope.go:117] "RemoveContainer" containerID="53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58" Jul 15 13:45:44.095987 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:44.095959 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:44.096105 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:44.096042 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:44.096105 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:44.096094 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:45.101744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:45.101705 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" event={"ID":"c1b6046b-b229-44dc-9694-4b9c920504f4","Type":"ContainerStarted","Data":"c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6"} Jul 15 13:45:45.999567 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:45.999460 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:45.999723 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:45.999609 2564 patch_prober.go:28] interesting pod/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 15 13:45:45.999723 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:45.999659 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 15 13:45:46.999386 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:46.999350 2564 patch_prober.go:28] interesting pod/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 15 13:45:46.999776 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:46.999411 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 15 13:45:47.999086 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:47.999045 2564 patch_prober.go:28] interesting pod/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 15 13:45:47.999275 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:47.999110 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 15 13:45:48.999107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:48.999073 2564 patch_prober.go:28] interesting pod/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" start-of-body= Jul 15 13:45:48.999562 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:48.999148 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.55:15021/healthz/ready\": dial tcp 10.134.0.55:15021: connect: connection refused" Jul 15 13:45:50.003063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:50.003035 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:50.003508 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:50.003494 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:50.004097 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:50.004079 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:45:50.137357 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:50.136804 2564 patch_prober.go:28] interesting pod/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:43142->10.134.0.56:15021: read: connection reset by peer" start-of-body= Jul 15 13:45:50.137357 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:50.136882 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": read tcp 10.134.0.2:43142->10.134.0.56:15021: read: connection reset by peer" Jul 15 13:45:51.136311 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:51.136275 2564 generic.go:356] "Generic (PLEG): container finished" podID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerID="9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7" exitCode=0 Jul 15 13:45:51.136817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:51.136347 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" event={"ID":"eaee3805-1407-4fff-bfc5-c821aef0eecd","Type":"ContainerDied","Data":"9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7"} Jul 15 13:45:51.136926 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:51.136909 2564 scope.go:117] "RemoveContainer" containerID="9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7" Jul 15 13:45:51.137864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:51.137838 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:51.137960 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:51.137904 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:51.137960 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:51.137935 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:45:52.144525 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:52.144486 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" event={"ID":"eaee3805-1407-4fff-bfc5-c821aef0eecd","Type":"ContainerStarted","Data":"c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8"} Jul 15 13:45:52.382792 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:52.382755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:52.382972 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:52.382953 2564 patch_prober.go:28] interesting pod/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 15 13:45:52.383030 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:52.382997 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 15 13:45:53.383462 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:53.383424 2564 patch_prober.go:28] interesting pod/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 15 13:45:53.383877 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:53.383502 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 15 13:45:54.383365 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:54.383320 2564 patch_prober.go:28] interesting pod/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 15 13:45:54.383565 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:54.383390 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 15 13:45:55.383655 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:55.383619 2564 patch_prober.go:28] interesting pod/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 15 13:45:55.384068 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:55.383680 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 15 13:45:56.383566 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:56.383533 2564 patch_prober.go:28] interesting pod/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body= Jul 15 13:45:56.383746 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:56.383583 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" Jul 15 13:45:57.387173 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:57.387143 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:57.387581 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:57.387426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:45:57.388158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:45:57.388136 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:46:34.738422 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:46:34.738395 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:46:34.750641 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:46:34.750618 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:47:11.017027 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:11.016948 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28"] Jul 15 13:47:11.017514 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:11.017190 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" containerID="cri-o://c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6" gracePeriod=30 Jul 15 13:47:16.266391 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.266368 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:47:16.414107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414071 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b6046b-b229-44dc-9694-4b9c920504f4-istiod-ca-cert\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414108 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xzk7w\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-kube-api-access-xzk7w\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414132 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-credential-socket\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414166 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-podinfo\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414196 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-token\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414222 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-certs\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414294 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-data\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414393 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-envoy\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414449 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-socket\") pod \"c1b6046b-b229-44dc-9694-4b9c920504f4\" (UID: \"c1b6046b-b229-44dc-9694-4b9c920504f4\") " Jul 15 13:47:16.414665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414507 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:16.414665 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b6046b-b229-44dc-9694-4b9c920504f4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:47:16.414875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414800 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:16.414875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414868 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1b6046b-b229-44dc-9694-4b9c920504f4-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.414988 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414889 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.414988 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.414859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:16.415383 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.415359 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:16.416458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.416426 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-kube-api-access-xzk7w" (OuterVolumeSpecName: "kube-api-access-xzk7w") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "kube-api-access-xzk7w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:47:16.416458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.416448 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:47:16.416864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.416837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:47:16.417048 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.417029 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1b6046b-b229-44dc-9694-4b9c920504f4" (UID: "c1b6046b-b229-44dc-9694-4b9c920504f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:16.515841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515803 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.515841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515837 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.515841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515846 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.515841 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515854 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.516106 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515862 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.516106 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515870 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1b6046b-b229-44dc-9694-4b9c920504f4-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.516106 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.515878 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzk7w\" (UniqueName: \"kubernetes.io/projected/c1b6046b-b229-44dc-9694-4b9c920504f4-kube-api-access-xzk7w\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:16.607654 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.607624 2564 generic.go:356] "Generic (PLEG): container finished" podID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerID="c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6" exitCode=0 Jul 15 13:47:16.607759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.607676 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" Jul 15 13:47:16.607759 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.607717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" event={"ID":"c1b6046b-b229-44dc-9694-4b9c920504f4","Type":"ContainerDied","Data":"c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6"} Jul 15 13:47:16.607840 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.607804 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28" event={"ID":"c1b6046b-b229-44dc-9694-4b9c920504f4","Type":"ContainerDied","Data":"1635eac0a19de4aa397a3633b1530af314cdda25b000f9f33d922ea768a7da72"} Jul 15 13:47:16.607840 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.607832 2564 scope.go:117] "RemoveContainer" containerID="c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6" Jul 15 13:47:16.617349 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.617333 2564 scope.go:117] "RemoveContainer" containerID="53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58" Jul 15 13:47:16.625153 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.625138 2564 scope.go:117] "RemoveContainer" containerID="c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6" Jul 15 13:47:16.625404 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:47:16.625386 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6\": container with ID starting with c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6 not found: ID does not exist" containerID="c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6" Jul 15 13:47:16.625480 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.625412 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6"} err="failed to get container status \"c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6\": rpc error: code = NotFound desc = could not find container \"c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6\": container with ID starting with c94e05a2ee894874b26633d51cd68076c9a9bbbf2888423ca279bab875a191a6 not found: ID does not exist" Jul 15 13:47:16.625480 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.625430 2564 scope.go:117] "RemoveContainer" containerID="53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58" Jul 15 13:47:16.625692 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:47:16.625677 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58\": container with ID starting with 53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58 not found: ID does not exist" containerID="53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58" Jul 15 13:47:16.625734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.625696 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58"} err="failed to get container status \"53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58\": rpc error: code = NotFound desc = could not find container \"53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58\": container with ID starting with 53f8f866ee1bd64085628ec0d0463617e0f432224c32d19f9d575eea9e3c3e58 not found: ID does not exist" Jul 15 13:47:16.629674 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.629655 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28"] Jul 15 13:47:16.633127 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.633108 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-486f14-openshift-default-b475d5b8c-mqk28"] Jul 15 13:47:16.657757 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:16.657739 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" path="/var/lib/kubelet/pods/c1b6046b-b229-44dc-9694-4b9c920504f4/volumes" Jul 15 13:47:37.127062 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.127025 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq"] Jul 15 13:47:37.127481 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.127273 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" containerID="cri-o://c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8" gracePeriod=30 Jul 15 13:47:37.985072 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985040 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr"] Jul 15 13:47:37.985463 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985450 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" Jul 15 13:47:37.985463 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985463 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" Jul 15 13:47:37.985618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985489 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" Jul 15 13:47:37.985618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985494 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" Jul 15 13:47:37.985618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985555 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" Jul 15 13:47:37.985618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.985567 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1b6046b-b229-44dc-9694-4b9c920504f4" containerName="istio-proxy" Jul 15 13:47:37.988950 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.988932 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:37.991678 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:37.991658 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-23e666-openshift-default-dockercfg-927hp\"" Jul 15 13:47:38.003210 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.003189 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr"] Jul 15 13:47:38.005694 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.005673 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-envoy\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.005785 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.005728 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-certs\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.005842 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.005827 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-credential-socket\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.005880 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.005867 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e5be77-c712-49be-9303-f07cf9552a9d-istio-podinfo\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.005945 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.005923 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e5be77-c712-49be-9303-f07cf9552a9d-istiod-ca-cert\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.006006 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.005967 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z62zc\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-kube-api-access-z62zc\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.006060 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.006022 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-data\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.006114 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.006062 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-socket\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.006166 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.006113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-istio-token\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107055 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107025 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-data\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-socket\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107096 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-istio-token\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107133 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-envoy\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107167 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-certs\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107218 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-credential-socket\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107506 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107313 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e5be77-c712-49be-9303-f07cf9552a9d-istio-podinfo\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107506 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107346 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e5be77-c712-49be-9303-f07cf9552a9d-istiod-ca-cert\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107506 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107376 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z62zc\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-kube-api-access-z62zc\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107506 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107482 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-data\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107703 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107533 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-socket\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107748 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107712 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-credential-socket\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.107914 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.107892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-certs\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.108152 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.108114 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e5be77-c712-49be-9303-f07cf9552a9d-istiod-ca-cert\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.109705 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.109676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-envoy\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.109891 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.109872 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e5be77-c712-49be-9303-f07cf9552a9d-istio-podinfo\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.119626 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.119602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-istio-token\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.119843 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.119820 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z62zc\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-kube-api-access-z62zc\") pod \"e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.300165 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.300137 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:38.439670 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.439645 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr"] Jul 15 13:47:38.441622 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:47:38.441593 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e5be77_c712_49be_9303_f07cf9552a9d.slice/crio-21b9f4651a365d53712e09d1b2797191791fd8d0125e213048d4f8d0e1c12192 WatchSource:0}: Error finding container 21b9f4651a365d53712e09d1b2797191791fd8d0125e213048d4f8d0e1c12192: Status 404 returned error can't find the container with id 21b9f4651a365d53712e09d1b2797191791fd8d0125e213048d4f8d0e1c12192 Jul 15 13:47:38.443781 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.443750 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:38.443855 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.443812 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:38.443855 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.443840 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:38.721788 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.721708 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" event={"ID":"88e5be77-c712-49be-9303-f07cf9552a9d","Type":"ContainerStarted","Data":"f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4"} Jul 15 13:47:38.721788 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.721767 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" event={"ID":"88e5be77-c712-49be-9303-f07cf9552a9d","Type":"ContainerStarted","Data":"21b9f4651a365d53712e09d1b2797191791fd8d0125e213048d4f8d0e1c12192"} Jul 15 13:47:38.744049 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:38.744003 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podStartSLOduration=1.743989058 podStartE2EDuration="1.743989058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:47:38.741954224 +0000 UTC m=+2464.697408822" watchObservedRunningTime="2026-07-15 13:47:38.743989058 +0000 UTC m=+2464.699443620" Jul 15 13:47:39.300835 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:39.300795 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:39.302295 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:39.302270 2564 patch_prober.go:28] interesting pod/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 15 13:47:39.302417 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:39.302319 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 15 13:47:40.305012 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:40.304980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:40.732667 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:40.732585 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:40.733500 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:40.733444 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:42.379769 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.379748 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:47:42.450141 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450073 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-certs\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450141 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450115 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-socket\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450141 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450132 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-credential-socket\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450158 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-token\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450198 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-podinfo\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450227 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-data\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450319 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-envoy\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450345 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaee3805-1407-4fff-bfc5-c821aef0eecd-istiod-ca-cert\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450358 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450373 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:42.450392 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450370 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6gd7q\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-kube-api-access-6gd7q\") pod \"eaee3805-1407-4fff-bfc5-c821aef0eecd\" (UID: \"eaee3805-1407-4fff-bfc5-c821aef0eecd\") " Jul 15 13:47:42.450838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450538 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:42.450838 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450729 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaee3805-1407-4fff-bfc5-c821aef0eecd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:47:42.450955 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450919 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eaee3805-1407-4fff-bfc5-c821aef0eecd-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.450955 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450939 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.450955 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450953 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.451115 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.450966 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.451399 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.451376 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-data" (OuterVolumeSpecName: "istio-data") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:42.452440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.452419 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:47:42.452609 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.452588 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:47:42.452857 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.452831 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-kube-api-access-6gd7q" (OuterVolumeSpecName: "kube-api-access-6gd7q") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "kube-api-access-6gd7q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:47:42.452950 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.452859 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-token" (OuterVolumeSpecName: "istio-token") pod "eaee3805-1407-4fff-bfc5-c821aef0eecd" (UID: "eaee3805-1407-4fff-bfc5-c821aef0eecd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:47:42.552240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.552206 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.552240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.552233 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.552240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.552242 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.552240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.552251 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eaee3805-1407-4fff-bfc5-c821aef0eecd-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.552543 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.552259 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gd7q\" (UniqueName: \"kubernetes.io/projected/eaee3805-1407-4fff-bfc5-c821aef0eecd-kube-api-access-6gd7q\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:47:42.743642 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.743552 2564 generic.go:356] "Generic (PLEG): container finished" podID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerID="c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8" exitCode=0 Jul 15 13:47:42.743642 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.743602 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" Jul 15 13:47:42.743862 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.743634 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" event={"ID":"eaee3805-1407-4fff-bfc5-c821aef0eecd","Type":"ContainerDied","Data":"c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8"} Jul 15 13:47:42.743862 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.743690 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq" event={"ID":"eaee3805-1407-4fff-bfc5-c821aef0eecd","Type":"ContainerDied","Data":"4951f87f7d5cb749e2000d1bf28b4059949ede8e9ac8dfe4e7322fe693c7e8e1"} Jul 15 13:47:42.743862 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.743715 2564 scope.go:117] "RemoveContainer" containerID="c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8" Jul 15 13:47:42.752895 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.752877 2564 scope.go:117] "RemoveContainer" containerID="9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7" Jul 15 13:47:42.760536 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.760515 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq"] Jul 15 13:47:42.761263 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.761246 2564 scope.go:117] "RemoveContainer" containerID="c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8" Jul 15 13:47:42.761582 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:47:42.761552 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8\": container with ID starting with c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8 not found: ID does not exist" containerID="c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8" Jul 15 13:47:42.761582 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.761587 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8"} err="failed to get container status \"c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8\": rpc error: code = NotFound desc = could not find container \"c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8\": container with ID starting with c03ad2d59bc097fbc779475df5089bd2c85e4b52cf2afbbe1c32182c25ce7ae8 not found: ID does not exist" Jul 15 13:47:42.761765 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.761612 2564 scope.go:117] "RemoveContainer" containerID="9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7" Jul 15 13:47:42.761980 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:47:42.761949 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7\": container with ID starting with 9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7 not found: ID does not exist" containerID="9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7" Jul 15 13:47:42.762122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.761984 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7"} err="failed to get container status \"9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7\": rpc error: code = NotFound desc = could not find container \"9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7\": container with ID starting with 9a9e5875c5d448937dacfd82c3eb674ea151bbfd01588dc22a7be9c1a788c3a7 not found: ID does not exist" Jul 15 13:47:42.764662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:42.764642 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-a5d1bc-openshift-default-7d6b68cf8c-8bjbq"] Jul 15 13:47:44.659009 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:44.658978 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" path="/var/lib/kubelet/pods/eaee3805-1407-4fff-bfc5-c821aef0eecd/volumes" Jul 15 13:47:45.870677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.870601 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp"] Jul 15 13:47:45.871221 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.871205 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" Jul 15 13:47:45.871269 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.871226 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" Jul 15 13:47:45.871367 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.871356 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" Jul 15 13:47:45.871402 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.871376 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" Jul 15 13:47:45.875575 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.875556 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.878161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.878142 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-929ef6-openshift-default-dockercfg-j9k4l\"" Jul 15 13:47:45.886022 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.886001 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp"] Jul 15 13:47:45.987259 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987226 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-token\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987278 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-data\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-credential-socket\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987366 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-podinfo\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6sr\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-kube-api-access-kx6sr\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-certs\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987492 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-envoy\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987511 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-socket\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:45.987624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:45.987532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istiod-ca-cert\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.088852 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.088815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-envoy\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.088852 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.088853 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-socket\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089080 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.088877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istiod-ca-cert\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089080 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.088929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-token\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089190 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089101 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-data\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089190 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089161 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-credential-socket\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089296 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-podinfo\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089296 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089247 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6sr\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-kube-api-access-kx6sr\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089296 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089277 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-certs\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089454 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089336 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-socket\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089581 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089557 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-credential-socket\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089676 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istiod-ca-cert\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089732 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089662 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-certs\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.089842 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.089819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-data\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.091209 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.091182 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-envoy\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.091459 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.091440 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-podinfo\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.096745 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.096723 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-token\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.096971 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.096954 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6sr\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-kube-api-access-kx6sr\") pod \"e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.186579 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.186486 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:46.323921 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.323882 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp"] Jul 15 13:47:46.325656 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:47:46.325628 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod432db5c4_a4b2_4ab0_af31_e5691d3e2b1e.slice/crio-cc70776c93ce575a9f36f44b5fea441b6ade9d3baaaf8567921d104ff156678e WatchSource:0}: Error finding container cc70776c93ce575a9f36f44b5fea441b6ade9d3baaaf8567921d104ff156678e: Status 404 returned error can't find the container with id cc70776c93ce575a9f36f44b5fea441b6ade9d3baaaf8567921d104ff156678e Jul 15 13:47:46.328254 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.328225 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:46.328337 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.328289 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:46.328337 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.328316 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:46.768484 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.768427 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" event={"ID":"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e","Type":"ContainerStarted","Data":"1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c"} Jul 15 13:47:46.768685 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.768522 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" event={"ID":"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e","Type":"ContainerStarted","Data":"cc70776c93ce575a9f36f44b5fea441b6ade9d3baaaf8567921d104ff156678e"} Jul 15 13:47:46.792918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:46.792851 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podStartSLOduration=1.79282915 podStartE2EDuration="1.79282915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:47:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:47:46.792690874 +0000 UTC m=+2472.748145551" watchObservedRunningTime="2026-07-15 13:47:46.79282915 +0000 UTC m=+2472.748283713" Jul 15 13:47:47.187426 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:47.187386 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:47.192400 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:47.192371 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:47.773654 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:47.773623 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:47.774793 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:47.774768 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:47:53.806875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:53.806847 2564 generic.go:356] "Generic (PLEG): container finished" podID="88e5be77-c712-49be-9303-f07cf9552a9d" containerID="f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4" exitCode=0 Jul 15 13:47:53.807244 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:53.806913 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" event={"ID":"88e5be77-c712-49be-9303-f07cf9552a9d","Type":"ContainerDied","Data":"f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4"} Jul 15 13:47:53.807318 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:53.807304 2564 scope.go:117] "RemoveContainer" containerID="f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4" Jul 15 13:47:53.808416 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:53.808382 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:53.808541 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:53.808451 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:53.808541 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:53.808529 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:47:54.813742 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:54.813710 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" event={"ID":"88e5be77-c712-49be-9303-f07cf9552a9d","Type":"ContainerStarted","Data":"6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85"} Jul 15 13:47:55.300570 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:55.300533 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:55.301768 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:55.301739 2564 patch_prober.go:28] interesting pod/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 15 13:47:55.301897 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:55.301800 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 15 13:47:55.732936 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:55.732837 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:56.301074 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:56.301034 2564 patch_prober.go:28] interesting pod/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 15 13:47:56.301458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:56.301102 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 15 13:47:57.301328 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:57.301291 2564 patch_prober.go:28] interesting pod/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 15 13:47:57.301734 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:57.301344 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 15 13:47:58.300554 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:58.300509 2564 patch_prober.go:28] interesting pod/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body= Jul 15 13:47:58.300771 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:58.300578 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" Jul 15 13:47:59.304671 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:59.304637 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:47:59.305654 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:47:59.305632 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:48:03.815682 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:03.815638 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 15 13:48:03.816076 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:03.815716 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 15 13:48:04.885738 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:04.885704 2564 generic.go:356] "Generic (PLEG): container finished" podID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerID="1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c" exitCode=0 Jul 15 13:48:04.886253 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:04.885793 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" event={"ID":"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e","Type":"ContainerDied","Data":"1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c"} Jul 15 13:48:04.886331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:04.886277 2564 scope.go:117] "RemoveContainer" containerID="1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c" Jul 15 13:48:04.887368 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:04.887340 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:48:04.887487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:04.887403 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:48:04.887487 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:04.887433 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:48:05.892593 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:05.892562 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" event={"ID":"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e","Type":"ContainerStarted","Data":"80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767"} Jul 15 13:48:06.187234 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:06.187142 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:48:06.187510 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:06.187463 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 15 13:48:06.187568 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:06.187534 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 15 13:48:07.187858 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:07.187823 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 15 13:48:07.188264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:07.187877 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 15 13:48:08.187807 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:08.187770 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 15 13:48:08.187989 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:08.187827 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 15 13:48:09.187798 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:09.187761 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 15 13:48:09.187987 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:09.187816 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 15 13:48:10.187032 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:10.186992 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body= Jul 15 13:48:10.187427 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:10.187054 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" Jul 15 13:48:11.190884 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:11.190845 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:48:11.191343 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:11.191295 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:48:11.191991 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:11.191971 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:48:26.964591 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:26.964492 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr"] Jul 15 13:48:26.965074 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:26.964734 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" containerID="cri-o://6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85" gracePeriod=30 Jul 15 13:48:29.307518 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:29.307434 2564 patch_prober.go:28] interesting pod/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Jul 15 13:48:29.307976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:29.307550 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:48:32.209631 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.209604 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:48:32.298754 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298722 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-certs\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.298924 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298762 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e5be77-c712-49be-9303-f07cf9552a9d-istio-podinfo\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.298924 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298778 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-istio-token\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.298924 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298797 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-credential-socket\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.298924 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298835 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-data\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.299102 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298936 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-socket\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.299102 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298969 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-envoy\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.299102 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.298989 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e5be77-c712-49be-9303-f07cf9552a9d-istiod-ca-cert\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.299102 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299082 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z62zc\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-kube-api-access-z62zc\") pod \"88e5be77-c712-49be-9303-f07cf9552a9d\" (UID: \"88e5be77-c712-49be-9303-f07cf9552a9d\") " Jul 15 13:48:32.299291 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299106 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:32.299291 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299141 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:32.299401 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299335 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:32.299606 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299540 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.299606 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299565 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.299606 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299569 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e5be77-c712-49be-9303-f07cf9552a9d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:48:32.299606 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299579 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.299917 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.299725 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-data" (OuterVolumeSpecName: "istio-data") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:32.301157 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.301134 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:32.301270 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.301168 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-istio-token" (OuterVolumeSpecName: "istio-token") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:48:32.301270 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.301220 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88e5be77-c712-49be-9303-f07cf9552a9d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:48:32.301348 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.301311 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-kube-api-access-z62zc" (OuterVolumeSpecName: "kube-api-access-z62zc") pod "88e5be77-c712-49be-9303-f07cf9552a9d" (UID: "88e5be77-c712-49be-9303-f07cf9552a9d"). InnerVolumeSpecName "kube-api-access-z62zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:48:32.401054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.401021 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.401054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.401048 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88e5be77-c712-49be-9303-f07cf9552a9d-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.401054 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.401056 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88e5be77-c712-49be-9303-f07cf9552a9d-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.401284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.401065 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z62zc\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-kube-api-access-z62zc\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.401284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.401075 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88e5be77-c712-49be-9303-f07cf9552a9d-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:32.401284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:32.401083 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88e5be77-c712-49be-9303-f07cf9552a9d-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:33.045321 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.045287 2564 generic.go:356] "Generic (PLEG): container finished" podID="88e5be77-c712-49be-9303-f07cf9552a9d" containerID="6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85" exitCode=0 Jul 15 13:48:33.045557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.045333 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" Jul 15 13:48:33.045557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.045378 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" event={"ID":"88e5be77-c712-49be-9303-f07cf9552a9d","Type":"ContainerDied","Data":"6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85"} Jul 15 13:48:33.045557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.045435 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr" event={"ID":"88e5be77-c712-49be-9303-f07cf9552a9d","Type":"ContainerDied","Data":"21b9f4651a365d53712e09d1b2797191791fd8d0125e213048d4f8d0e1c12192"} Jul 15 13:48:33.045557 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.045457 2564 scope.go:117] "RemoveContainer" containerID="6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85" Jul 15 13:48:33.054760 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.054745 2564 scope.go:117] "RemoveContainer" containerID="f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4" Jul 15 13:48:33.063316 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.063300 2564 scope.go:117] "RemoveContainer" containerID="6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85" Jul 15 13:48:33.063621 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:48:33.063595 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85\": container with ID starting with 6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85 not found: ID does not exist" containerID="6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85" Jul 15 13:48:33.063694 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.063629 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85"} err="failed to get container status \"6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85\": rpc error: code = NotFound desc = could not find container \"6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85\": container with ID starting with 6da1f3cac2c2a776b13784c80aee3f7fa340c7533290559e6e0b7eaaa2ed4c85 not found: ID does not exist" Jul 15 13:48:33.063694 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.063648 2564 scope.go:117] "RemoveContainer" containerID="f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4" Jul 15 13:48:33.063909 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:48:33.063887 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4\": container with ID starting with f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4 not found: ID does not exist" containerID="f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4" Jul 15 13:48:33.063961 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.063921 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4"} err="failed to get container status \"f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4\": rpc error: code = NotFound desc = could not find container \"f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4\": container with ID starting with f259dffb46a0e3e6d971be31055412fb96d2df22e109eea23245a8a9d781aad4 not found: ID does not exist" Jul 15 13:48:33.070084 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.070061 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr"] Jul 15 13:48:33.072275 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:33.072255 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-23e666-openshift-default-6d96b88f7f-lhgfr"] Jul 15 13:48:34.665000 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:34.664959 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" path="/var/lib/kubelet/pods/88e5be77-c712-49be-9303-f07cf9552a9d/volumes" Jul 15 13:48:53.094324 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.094290 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp"] Jul 15 13:48:53.094797 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.094548 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" containerID="cri-o://80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767" gracePeriod=30 Jul 15 13:48:53.972816 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.972780 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j"] Jul 15 13:48:53.973330 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973310 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" Jul 15 13:48:53.973330 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973332 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" Jul 15 13:48:53.973498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973345 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" Jul 15 13:48:53.973498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973352 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" Jul 15 13:48:53.973498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973365 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" Jul 15 13:48:53.973498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973374 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="eaee3805-1407-4fff-bfc5-c821aef0eecd" containerName="istio-proxy" Jul 15 13:48:53.973701 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.973525 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" Jul 15 13:48:53.976899 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.976876 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:53.979399 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.979375 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-7abf62-openshift-default-dockercfg-6pghh\"" Jul 15 13:48:53.990573 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:53.990547 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j"] Jul 15 13:48:54.021481 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-socket\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021684 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021554 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-token\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021684 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08af9f30-decd-4e95-8229-cf61cddfee0e-istiod-ca-cert\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021684 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021664 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-envoy\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021697 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-data\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4m45\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-kube-api-access-w4m45\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-certs\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021859 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-podinfo\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.021923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.021886 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-credential-socket\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.123541 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123498 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-socket\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123677 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-token\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123742 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08af9f30-decd-4e95-8229-cf61cddfee0e-istiod-ca-cert\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-envoy\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123792 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-data\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4m45\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-kube-api-access-w4m45\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-certs\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123884 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-podinfo\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.123929 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-credential-socket\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.124159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-data\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.124272 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-socket\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.124298 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-certs\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124458 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.124365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-credential-socket\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.124740 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.124719 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08af9f30-decd-4e95-8229-cf61cddfee0e-istiod-ca-cert\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.126011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.125986 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-envoy\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.126406 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.126389 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-podinfo\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.132420 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.132400 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4m45\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-kube-api-access-w4m45\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.132550 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.132441 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-token\") pod \"e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.290611 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.290581 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:54.443935 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.443906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j"] Jul 15 13:48:54.445703 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:48:54.445673 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08af9f30_decd_4e95_8229_cf61cddfee0e.slice/crio-68d47b5648043287b0789737aa0a75d2d4e411eb78ece3f6e4050309fc3341cf WatchSource:0}: Error finding container 68d47b5648043287b0789737aa0a75d2d4e411eb78ece3f6e4050309fc3341cf: Status 404 returned error can't find the container with id 68d47b5648043287b0789737aa0a75d2d4e411eb78ece3f6e4050309fc3341cf Jul 15 13:48:54.448341 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.448310 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:48:54.448434 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.448374 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:48:54.448434 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:54.448402 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:48:55.164077 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:55.164041 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" event={"ID":"08af9f30-decd-4e95-8229-cf61cddfee0e","Type":"ContainerStarted","Data":"57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119"} Jul 15 13:48:55.164580 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:55.164096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" event={"ID":"08af9f30-decd-4e95-8229-cf61cddfee0e","Type":"ContainerStarted","Data":"68d47b5648043287b0789737aa0a75d2d4e411eb78ece3f6e4050309fc3341cf"} Jul 15 13:48:55.186810 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:55.186753 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podStartSLOduration=2.186736036 podStartE2EDuration="2.186736036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:48:55.184663167 +0000 UTC m=+2541.140117730" watchObservedRunningTime="2026-07-15 13:48:55.186736036 +0000 UTC m=+2541.142190598" Jul 15 13:48:55.291792 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:55.291757 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:55.299999 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:55.299973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:56.169242 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:56.169201 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:56.170070 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:56.170052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:48:56.193802 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:56.193769 2564 patch_prober.go:28] interesting pod/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 13:48:56.193943 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:56.193823 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:48:58.343636 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.343609 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:48:58.467041 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.466947 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-credential-socket\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467041 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467036 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-certs\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467229 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467072 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istiod-ca-cert\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467229 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467098 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx6sr\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-kube-api-access-kx6sr\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467229 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467163 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-data\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467229 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467198 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-envoy\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467234 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-podinfo\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467232 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:58.467440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467270 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-token\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467308 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-socket\") pod \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\" (UID: \"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e\") " Jul 15 13:48:58.467440 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467372 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:58.467754 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467486 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:48:58.467754 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467709 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:58.467859 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467843 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.467912 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467862 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.467912 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467877 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.467912 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.467894 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.468122 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.468087 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-data" (OuterVolumeSpecName: "istio-data") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:58.469502 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.469460 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:48:58.469502 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.469483 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-token" (OuterVolumeSpecName: "istio-token") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:48:58.469733 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.469710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:48:58.469834 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.469815 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-kube-api-access-kx6sr" (OuterVolumeSpecName: "kube-api-access-kx6sr") pod "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" (UID: "432db5c4-a4b2-4ab0-af31-e5691d3e2b1e"). InnerVolumeSpecName "kube-api-access-kx6sr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:48:58.569356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.569302 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6sr\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-kube-api-access-kx6sr\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.569356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.569357 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.569601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.569368 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.569601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.569376 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:58.569601 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:58.569384 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:48:59.188168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.188135 2564 generic.go:356] "Generic (PLEG): container finished" podID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerID="80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767" exitCode=0 Jul 15 13:48:59.188340 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.188183 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" Jul 15 13:48:59.188340 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.188217 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" event={"ID":"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e","Type":"ContainerDied","Data":"80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767"} Jul 15 13:48:59.188340 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.188273 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp" event={"ID":"432db5c4-a4b2-4ab0-af31-e5691d3e2b1e","Type":"ContainerDied","Data":"cc70776c93ce575a9f36f44b5fea441b6ade9d3baaaf8567921d104ff156678e"} Jul 15 13:48:59.188340 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.188296 2564 scope.go:117] "RemoveContainer" containerID="80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767" Jul 15 13:48:59.197757 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.197741 2564 scope.go:117] "RemoveContainer" containerID="1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c" Jul 15 13:48:59.206036 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.206004 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp"] Jul 15 13:48:59.206732 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.206715 2564 scope.go:117] "RemoveContainer" containerID="80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767" Jul 15 13:48:59.207083 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:48:59.207063 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767\": container with ID starting with 80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767 not found: ID does not exist" containerID="80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767" Jul 15 13:48:59.207132 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.207096 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767"} err="failed to get container status \"80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767\": rpc error: code = NotFound desc = could not find container \"80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767\": container with ID starting with 80b8e695cbc026ba3ca2505000c36540f200efa0fb52d8e1a2f23dd6dc34f767 not found: ID does not exist" Jul 15 13:48:59.207132 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.207115 2564 scope.go:117] "RemoveContainer" containerID="1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c" Jul 15 13:48:59.207367 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:48:59.207340 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c\": container with ID starting with 1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c not found: ID does not exist" containerID="1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c" Jul 15 13:48:59.207415 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.207378 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c"} err="failed to get container status \"1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c\": rpc error: code = NotFound desc = could not find container \"1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c\": container with ID starting with 1f340e499abd0a268b8f60748590e66c27f003935d98f3aacc4c98cee457850c not found: ID does not exist" Jul 15 13:48:59.211232 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:48:59.211211 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-929ef6-openshift-default-5d6bdb69cf-xcwbp"] Jul 15 13:49:00.658286 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:00.658250 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" path="/var/lib/kubelet/pods/432db5c4-a4b2-4ab0-af31-e5691d3e2b1e/volumes" Jul 15 13:49:02.212350 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.212317 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg"] Jul 15 13:49:02.213011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.212994 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" Jul 15 13:49:02.213063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.213014 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" Jul 15 13:49:02.213063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.213040 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" Jul 15 13:49:02.213063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.213048 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" Jul 15 13:49:02.213161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.213135 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" Jul 15 13:49:02.213161 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.213147 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="432db5c4-a4b2-4ab0-af31-e5691d3e2b1e" containerName="istio-proxy" Jul 15 13:49:02.213229 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.213164 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e5be77-c712-49be-9303-f07cf9552a9d" containerName="istio-proxy" Jul 15 13:49:02.218357 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.218337 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.220715 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.220696 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-bc6fa6-openshift-default-dockercfg-ptb96\"" Jul 15 13:49:02.226137 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.226103 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg"] Jul 15 13:49:02.305041 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305002 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-envoy\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305117 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-data\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305146 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-credential-socket\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-token\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-podinfo\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305240 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-socket\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305580 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305321 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istiod-ca-cert\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305580 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305369 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-certs\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.305580 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.305408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq8kr\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-kube-api-access-fq8kr\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406689 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406648 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-envoy\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-data\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406754 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-credential-socket\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-token\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-podinfo\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406813 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-socket\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.406864 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406854 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istiod-ca-cert\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.406886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-certs\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.407204 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-credential-socket\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407228 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.407211 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-data\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407384 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.407347 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq8kr\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-kube-api-access-fq8kr\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407518 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.407493 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-certs\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407719 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.407698 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-socket\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.407847 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.407823 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istiod-ca-cert\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.409002 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.408975 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-envoy\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.409342 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.409323 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-podinfo\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.415725 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.415702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-token\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.415823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.415805 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq8kr\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-kube-api-access-fq8kr\") pod \"e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.554076 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.554031 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:02.721245 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.721219 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg"] Jul 15 13:49:02.722788 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:49:02.722744 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6363c76f_ff3e_4e9e_8951_e9e8d1232dd8.slice/crio-8b3b4cd68649e4d2aaac9debcd65d99c82e4b449e9dd0372d66b16c4afe61b5d WatchSource:0}: Error finding container 8b3b4cd68649e4d2aaac9debcd65d99c82e4b449e9dd0372d66b16c4afe61b5d: Status 404 returned error can't find the container with id 8b3b4cd68649e4d2aaac9debcd65d99c82e4b449e9dd0372d66b16c4afe61b5d Jul 15 13:49:02.725064 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.725027 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:02.725174 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.725091 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:02.725174 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:02.725121 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:03.212883 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:03.212850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" event={"ID":"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8","Type":"ContainerStarted","Data":"d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927"} Jul 15 13:49:03.213354 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:03.212900 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" event={"ID":"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8","Type":"ContainerStarted","Data":"8b3b4cd68649e4d2aaac9debcd65d99c82e4b449e9dd0372d66b16c4afe61b5d"} Jul 15 13:49:03.555168 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:03.555075 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:03.556934 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:03.556655 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 15 13:49:03.556934 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:03.556715 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 15 13:49:04.558428 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:04.558397 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:04.578928 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:04.578879 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podStartSLOduration=2.578863674 podStartE2EDuration="2.578863674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:49:03.239024242 +0000 UTC m=+2549.194478806" watchObservedRunningTime="2026-07-15 13:49:04.578863674 +0000 UTC m=+2550.534318237" Jul 15 13:49:05.223032 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:05.222998 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:05.224078 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:05.224055 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:09.246217 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:09.246157 2564 generic.go:356] "Generic (PLEG): container finished" podID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerID="57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119" exitCode=0 Jul 15 13:49:09.246599 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:09.246231 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" event={"ID":"08af9f30-decd-4e95-8229-cf61cddfee0e","Type":"ContainerDied","Data":"57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119"} Jul 15 13:49:09.246646 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:09.246635 2564 scope.go:117] "RemoveContainer" containerID="57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119" Jul 15 13:49:09.247650 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:09.247620 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:09.247746 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:09.247685 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:09.247746 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:09.247736 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:10.255017 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:10.254982 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" event={"ID":"08af9f30-decd-4e95-8229-cf61cddfee0e","Type":"ContainerStarted","Data":"955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75"} Jul 15 13:49:10.291542 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:10.291511 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:49:10.292381 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:10.292354 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 13:49:10.292507 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:10.292412 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 13:49:11.169529 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:11.169457 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:49:11.291626 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:11.291589 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 13:49:11.291981 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:11.291645 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 13:49:12.291261 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:12.291228 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 13:49:12.291423 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:12.291310 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 13:49:13.291897 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:13.291863 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 13:49:13.292356 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:13.292325 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 13:49:14.291257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:14.291223 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" start-of-body= Jul 15 13:49:14.291443 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:14.291291 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.59:15021/healthz/ready\": dial tcp 10.134.0.59:15021: connect: connection refused" Jul 15 13:49:15.295273 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:15.295241 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:49:15.296338 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:15.296320 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:49:20.224866 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.224774 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 15 13:49:20.224866 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.224844 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 15 13:49:20.318130 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.318093 2564 generic.go:356] "Generic (PLEG): container finished" podID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerID="d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927" exitCode=0 Jul 15 13:49:20.318305 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.318169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" event={"ID":"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8","Type":"ContainerDied","Data":"d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927"} Jul 15 13:49:20.318720 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.318697 2564 scope.go:117] "RemoveContainer" containerID="d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927" Jul 15 13:49:20.319860 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.319830 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:20.319958 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.319896 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:20.319958 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:20.319926 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"} Jul 15 13:49:21.326071 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:21.326038 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" event={"ID":"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8","Type":"ContainerStarted","Data":"4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb"} Jul 15 13:49:21.554491 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:21.554442 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:21.554835 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:21.554813 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 15 13:49:21.555011 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:21.554984 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 15 13:49:22.554746 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:22.554704 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 15 13:49:22.555233 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:22.554772 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 15 13:49:23.555545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:23.555231 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 15 13:49:23.555545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:23.555293 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 15 13:49:24.555513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:24.555364 2564 patch_prober.go:28] interesting pod/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Jul 15 13:49:24.555513 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:24.555436 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Jul 15 13:49:25.558573 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:25.558541 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:25.559035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:25.558854 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:25.559561 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:25.559545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:49:48.999265 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:48.999220 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g"] Jul 15 13:49:49.003447 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.003429 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.007247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.007006 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"kube-root-ca.crt\"" Jul 15 13:49:49.007247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.007077 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"default-dockercfg-gx7v7\"" Jul 15 13:49:49.007247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.007087 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"openshift-service-ca.crt\"" Jul 15 13:49:49.007247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.007138 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"auth-test-model-bc6fa6-kserve-self-signed-certs\"" Jul 15 13:49:49.010291 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.010272 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g"] Jul 15 13:49:49.090047 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.090004 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-home\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.090047 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.090049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-model-cache\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.090315 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.090159 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-tmp-dir\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.090315 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.090210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-dshm\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.090315 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.090231 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q89mp\" (UniqueName: \"kubernetes.io/projected/affbcf42-a938-4361-be45-d6841bcbde45-kube-api-access-q89mp\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.090596 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.090310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affbcf42-a938-4361-be45-d6841bcbde45-tls-certs\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.191629 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.191593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-dshm\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.191794 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.191643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q89mp\" (UniqueName: \"kubernetes.io/projected/affbcf42-a938-4361-be45-d6841bcbde45-kube-api-access-q89mp\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.191794 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.191688 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affbcf42-a938-4361-be45-d6841bcbde45-tls-certs\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.191911 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.191783 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-home\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.191911 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.191849 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-model-cache\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.192013 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.191939 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-tmp-dir\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.192222 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.192198 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-home\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.192335 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.192241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-model-cache\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.192551 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.192531 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-tmp-dir\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.194618 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.194588 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affbcf42-a938-4361-be45-d6841bcbde45-tls-certs\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.194880 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.194854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-dshm\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.200301 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.200277 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q89mp\" (UniqueName: \"kubernetes.io/projected/affbcf42-a938-4361-be45-d6841bcbde45-kube-api-access-q89mp\") pod \"auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.313956 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.313917 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:49.447671 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.447643 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g"] Jul 15 13:49:49.449348 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:49:49.449324 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaffbcf42_a938_4361_be45_d6841bcbde45.slice/crio-81cda3ed69283a2a34a9cece909a0b23977321c0e3f71a5d4bd26f6841ec249d WatchSource:0}: Error finding container 81cda3ed69283a2a34a9cece909a0b23977321c0e3f71a5d4bd26f6841ec249d: Status 404 returned error can't find the container with id 81cda3ed69283a2a34a9cece909a0b23977321c0e3f71a5d4bd26f6841ec249d Jul 15 13:49:49.493113 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:49.493086 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" event={"ID":"affbcf42-a938-4361-be45-d6841bcbde45","Type":"ContainerStarted","Data":"81cda3ed69283a2a34a9cece909a0b23977321c0e3f71a5d4bd26f6841ec249d"} Jul 15 13:49:50.500159 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:50.500117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" event={"ID":"affbcf42-a938-4361-be45-d6841bcbde45","Type":"ContainerStarted","Data":"6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626"} Jul 15 13:49:50.519533 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:50.519486 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" podStartSLOduration=2.519458724 podStartE2EDuration="2.519458724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:49:50.516519636 +0000 UTC m=+2596.471974201" watchObservedRunningTime="2026-07-15 13:49:50.519458724 +0000 UTC m=+2596.474913286" Jul 15 13:49:59.315071 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:59.314973 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:59.315071 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:59.315048 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:59.327693 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:59.327664 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:49:59.564200 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:49:59.564171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:51:34.772647 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:51:34.772568 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:51:34.785327 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:51:34.785299 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:52:02.475830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:02.475799 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g"] Jul 15 13:52:17.496363 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.496298 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" podUID="affbcf42-a938-4361-be45-d6841bcbde45" containerName="main" containerID="cri-o://6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626" gracePeriod=45 Jul 15 13:52:17.747293 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.747234 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:52:17.854754 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.854725 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-dshm\") pod \"affbcf42-a938-4361-be45-d6841bcbde45\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " Jul 15 13:52:17.854923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.854785 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-tmp-dir\") pod \"affbcf42-a938-4361-be45-d6841bcbde45\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " Jul 15 13:52:17.854923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.854811 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-model-cache\") pod \"affbcf42-a938-4361-be45-d6841bcbde45\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " Jul 15 13:52:17.854923 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.854916 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-home\") pod \"affbcf42-a938-4361-be45-d6841bcbde45\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " Jul 15 13:52:17.855065 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.854951 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affbcf42-a938-4361-be45-d6841bcbde45-tls-certs\") pod \"affbcf42-a938-4361-be45-d6841bcbde45\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " Jul 15 13:52:17.855065 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.854992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q89mp\" (UniqueName: \"kubernetes.io/projected/affbcf42-a938-4361-be45-d6841bcbde45-kube-api-access-q89mp\") pod \"affbcf42-a938-4361-be45-d6841bcbde45\" (UID: \"affbcf42-a938-4361-be45-d6841bcbde45\") " Jul 15 13:52:17.855065 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.855036 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "affbcf42-a938-4361-be45-d6841bcbde45" (UID: "affbcf42-a938-4361-be45-d6841bcbde45"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:17.855217 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.855078 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-model-cache" (OuterVolumeSpecName: "model-cache") pod "affbcf42-a938-4361-be45-d6841bcbde45" (UID: "affbcf42-a938-4361-be45-d6841bcbde45"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:17.855217 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.855176 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-home" (OuterVolumeSpecName: "home") pod "affbcf42-a938-4361-be45-d6841bcbde45" (UID: "affbcf42-a938-4361-be45-d6841bcbde45"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:17.855367 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.855349 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-home\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.855419 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.855368 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-tmp-dir\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.855419 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.855383 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-model-cache\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.857008 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.856988 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-dshm" (OuterVolumeSpecName: "dshm") pod "affbcf42-a938-4361-be45-d6841bcbde45" (UID: "affbcf42-a938-4361-be45-d6841bcbde45"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:52:17.857077 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.857035 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/affbcf42-a938-4361-be45-d6841bcbde45-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "affbcf42-a938-4361-be45-d6841bcbde45" (UID: "affbcf42-a938-4361-be45-d6841bcbde45"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:52:17.857174 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.857141 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/affbcf42-a938-4361-be45-d6841bcbde45-kube-api-access-q89mp" (OuterVolumeSpecName: "kube-api-access-q89mp") pod "affbcf42-a938-4361-be45-d6841bcbde45" (UID: "affbcf42-a938-4361-be45-d6841bcbde45"). InnerVolumeSpecName "kube-api-access-q89mp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:52:17.955959 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.955931 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/affbcf42-a938-4361-be45-d6841bcbde45-dshm\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.955959 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.955957 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/affbcf42-a938-4361-be45-d6841bcbde45-tls-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:52:17.956098 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:17.955973 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q89mp\" (UniqueName: \"kubernetes.io/projected/affbcf42-a938-4361-be45-d6841bcbde45-kube-api-access-q89mp\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:52:18.281826 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.281799 2564 generic.go:356] "Generic (PLEG): container finished" podID="affbcf42-a938-4361-be45-d6841bcbde45" containerID="6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626" exitCode=0 Jul 15 13:52:18.281983 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.281843 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" Jul 15 13:52:18.281983 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.281855 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" event={"ID":"affbcf42-a938-4361-be45-d6841bcbde45","Type":"ContainerDied","Data":"6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626"} Jul 15 13:52:18.281983 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.281886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g" event={"ID":"affbcf42-a938-4361-be45-d6841bcbde45","Type":"ContainerDied","Data":"81cda3ed69283a2a34a9cece909a0b23977321c0e3f71a5d4bd26f6841ec249d"} Jul 15 13:52:18.281983 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.281902 2564 scope.go:117] "RemoveContainer" containerID="6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626" Jul 15 13:52:18.291201 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.291185 2564 scope.go:117] "RemoveContainer" containerID="6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626" Jul 15 13:52:18.291450 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:52:18.291431 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626\": container with ID starting with 6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626 not found: ID does not exist" containerID="6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626" Jul 15 13:52:18.291546 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.291458 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626"} err="failed to get container status \"6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626\": rpc error: code = NotFound desc = could not find container \"6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626\": container with ID starting with 6d202d20bde8b8c3e6d76d6c5cd61fcaf5a8ee9961710fc0ab0d0601f99ef626 not found: ID does not exist" Jul 15 13:52:18.302766 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.302746 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g"] Jul 15 13:52:18.306047 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.306023 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/auth-test-model-bc6fa6-kserve-58b77cb8d7-5cq5g"] Jul 15 13:52:18.657521 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:18.657493 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="affbcf42-a938-4361-be45-d6841bcbde45" path="/var/lib/kubelet/pods/affbcf42-a938-4361-be45-d6841bcbde45/volumes" Jul 15 13:52:43.999379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:43.999344 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5"] Jul 15 13:52:43.999883 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:43.999866 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="affbcf42-a938-4361-be45-d6841bcbde45" containerName="main" Jul 15 13:52:43.999928 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:43.999886 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="affbcf42-a938-4361-be45-d6841bcbde45" containerName="main" Jul 15 13:52:43.999964 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:43.999958 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="affbcf42-a938-4361-be45-d6841bcbde45" containerName="main" Jul 15 13:52:44.003032 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.003015 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.005158 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.005135 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"kube-root-ca.crt\"" Jul 15 13:52:44.006153 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.006135 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"default-dockercfg-gx7v7\"" Jul 15 13:52:44.006220 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.006197 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"openshift-service-ca.crt\"" Jul 15 13:52:44.006275 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.006219 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"sub-test-model-bc6fa6-kserve-self-signed-certs\"" Jul 15 13:52:44.012456 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.012437 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5"] Jul 15 13:52:44.093015 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.092981 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-home\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.093015 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.093019 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tls-certs\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.093226 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.093040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-dshm\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.093226 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.093097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-model-cache\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.093226 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.093135 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tmp-dir\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.093226 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.093157 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jsv\" (UniqueName: \"kubernetes.io/projected/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-kube-api-access-k4jsv\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.194630 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.194594 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-home\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.194630 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.194631 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tls-certs\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.194863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.194654 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-dshm\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.194863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.194694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-model-cache\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.194863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.194740 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tmp-dir\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.194863 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.194777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jsv\" (UniqueName: \"kubernetes.io/projected/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-kube-api-access-k4jsv\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.195109 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.195081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-home\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.195182 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.195095 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-model-cache\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.195182 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.195154 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tmp-dir\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.196878 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.196859 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-dshm\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.197170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.197152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tls-certs\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.203317 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.203294 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jsv\" (UniqueName: \"kubernetes.io/projected/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-kube-api-access-k4jsv\") pod \"sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.313433 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.313401 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:44.451005 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:44.450975 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5"] Jul 15 13:52:44.451885 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:52:44.451859 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode37bdad6_8fd9_4d88_89d2_9cbc8e77f53c.slice/crio-541fc9b407e260c2fe68c5a157ae0b6a9afc1ebda2d55ddee26291759f81e712 WatchSource:0}: Error finding container 541fc9b407e260c2fe68c5a157ae0b6a9afc1ebda2d55ddee26291759f81e712: Status 404 returned error can't find the container with id 541fc9b407e260c2fe68c5a157ae0b6a9afc1ebda2d55ddee26291759f81e712 Jul 15 13:52:45.442298 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:45.442257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" event={"ID":"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c","Type":"ContainerStarted","Data":"9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa"} Jul 15 13:52:45.442298 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:45.442300 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" event={"ID":"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c","Type":"ContainerStarted","Data":"541fc9b407e260c2fe68c5a157ae0b6a9afc1ebda2d55ddee26291759f81e712"} Jul 15 13:52:45.460146 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:45.460103 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" podStartSLOduration=2.460087647 podStartE2EDuration="2.460087647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:52:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:52:45.458671494 +0000 UTC m=+2771.414126110" watchObservedRunningTime="2026-07-15 13:52:45.460087647 +0000 UTC m=+2771.415542210" Jul 15 13:52:54.313927 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:54.313882 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:54.314425 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:54.313944 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:54.326272 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:54.326246 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:52:54.503647 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:52:54.503613 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:53:39.811860 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:39.811831 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5"] Jul 15 13:53:54.832272 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:54.832211 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" podUID="e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" containerName="main" containerID="cri-o://9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa" gracePeriod=45 Jul 15 13:53:55.079785 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.079761 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:53:55.180265 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180193 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-home\") pod \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " Jul 15 13:53:55.180265 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180257 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-model-cache\") pod \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " Jul 15 13:53:55.180455 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180306 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tls-certs\") pod \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " Jul 15 13:53:55.180455 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180325 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-dshm\") pod \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " Jul 15 13:53:55.180455 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180354 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k4jsv\" (UniqueName: \"kubernetes.io/projected/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-kube-api-access-k4jsv\") pod \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " Jul 15 13:53:55.180455 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180415 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tmp-dir\") pod \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\" (UID: \"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c\") " Jul 15 13:53:55.180718 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180501 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-home" (OuterVolumeSpecName: "home") pod "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" (UID: "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:55.180718 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180558 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-model-cache" (OuterVolumeSpecName: "model-cache") pod "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" (UID: "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:55.180815 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180719 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" (UID: "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:55.180875 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180856 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tmp-dir\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:53:55.180924 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180880 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-home\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:53:55.180924 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.180891 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-model-cache\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:53:55.182441 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.182415 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-dshm" (OuterVolumeSpecName: "dshm") pod "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" (UID: "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:53:55.182441 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.182432 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-kube-api-access-k4jsv" (OuterVolumeSpecName: "kube-api-access-k4jsv") pod "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" (UID: "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c"). InnerVolumeSpecName "kube-api-access-k4jsv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:53:55.182624 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.182463 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" (UID: "e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:53:55.282107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.282083 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-tls-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:53:55.282107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.282102 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-dshm\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:53:55.282107 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.282112 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4jsv\" (UniqueName: \"kubernetes.io/projected/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c-kube-api-access-k4jsv\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:53:55.831561 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.831527 2564 generic.go:356] "Generic (PLEG): container finished" podID="e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" containerID="9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa" exitCode=0 Jul 15 13:53:55.831728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.831566 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" Jul 15 13:53:55.831728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.831579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" event={"ID":"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c","Type":"ContainerDied","Data":"9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa"} Jul 15 13:53:55.831728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.831616 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5" event={"ID":"e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c","Type":"ContainerDied","Data":"541fc9b407e260c2fe68c5a157ae0b6a9afc1ebda2d55ddee26291759f81e712"} Jul 15 13:53:55.831728 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.831636 2564 scope.go:117] "RemoveContainer" containerID="9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa" Jul 15 13:53:55.851200 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.851040 2564 scope.go:117] "RemoveContainer" containerID="9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa" Jul 15 13:53:55.851497 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:53:55.851320 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa\": container with ID starting with 9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa not found: ID does not exist" containerID="9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa" Jul 15 13:53:55.851497 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.851345 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa"} err="failed to get container status \"9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa\": rpc error: code = NotFound desc = could not find container \"9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa\": container with ID starting with 9b524ab19531ec8227910f81f7712deee2aa01e5d5dbf30ee8e3d72157c95daa not found: ID does not exist" Jul 15 13:53:55.861089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.861065 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5"] Jul 15 13:53:55.866840 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:55.866810 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/sub-test-model-bc6fa6-kserve-d9d5f8755-v9xk5"] Jul 15 13:53:56.657486 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:53:56.657441 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" path="/var/lib/kubelet/pods/e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c/volumes" Jul 15 13:54:30.798940 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.798847 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn"] Jul 15 13:54:30.801271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.799277 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" containerName="main" Jul 15 13:54:30.801271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.799289 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" containerName="main" Jul 15 13:54:30.801271 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.799353 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e37bdad6-8fd9-4d88-89d2-9cbc8e77f53c" containerName="main" Jul 15 13:54:30.802358 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.802343 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:30.805483 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.805447 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"rate-test-model-bc6fa6-kserve-self-signed-certs\"" Jul 15 13:54:30.805593 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.805449 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"kube-root-ca.crt\"" Jul 15 13:54:30.805593 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.805449 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"default-dockercfg-gx7v7\"" Jul 15 13:54:30.805593 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.805526 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-bc6fa6\"/\"openshift-service-ca.crt\"" Jul 15 13:54:30.812262 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.812240 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn"] Jul 15 13:54:30.919720 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.919690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-home\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:30.919720 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.919721 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-dshm\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:30.919929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.919764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-model-cache\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:30.919929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.919809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ce0213-c081-4615-98ed-0ab2794bbc31-tls-certs\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:30.919929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.919825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-tmp-dir\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:30.919929 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:30.919887 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8s89\" (UniqueName: \"kubernetes.io/projected/d6ce0213-c081-4615-98ed-0ab2794bbc31-kube-api-access-c8s89\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.023280 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.023245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8s89\" (UniqueName: \"kubernetes.io/projected/d6ce0213-c081-4615-98ed-0ab2794bbc31-kube-api-access-c8s89\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.023541 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.023521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-home\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.023631 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.023558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-dshm\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.023698 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.023633 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-model-cache\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.023698 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.023672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ce0213-c081-4615-98ed-0ab2794bbc31-tls-certs\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.023807 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.023697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-tmp-dir\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.024089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.024055 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-model-cache\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.024089 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.024069 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-home\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.024231 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.024113 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-tmp-dir\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.026103 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.026081 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ce0213-c081-4615-98ed-0ab2794bbc31-tls-certs\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.026208 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.026165 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-dshm\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.031451 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.031429 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8s89\" (UniqueName: \"kubernetes.io/projected/d6ce0213-c081-4615-98ed-0ab2794bbc31-kube-api-access-c8s89\") pod \"rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.112313 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.112225 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:31.245035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:31.245012 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn"] Jul 15 13:54:31.247270 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:54:31.247243 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6ce0213_c081_4615_98ed_0ab2794bbc31.slice/crio-93aba789ec1fc723aa7a2ce6661f26eb322c0822e182df3771cd071c5b3b3b24 WatchSource:0}: Error finding container 93aba789ec1fc723aa7a2ce6661f26eb322c0822e182df3771cd071c5b3b3b24: Status 404 returned error can't find the container with id 93aba789ec1fc723aa7a2ce6661f26eb322c0822e182df3771cd071c5b3b3b24 Jul 15 13:54:32.029281 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:32.029243 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" event={"ID":"d6ce0213-c081-4615-98ed-0ab2794bbc31","Type":"ContainerStarted","Data":"31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25"} Jul 15 13:54:32.029677 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:32.029289 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" event={"ID":"d6ce0213-c081-4615-98ed-0ab2794bbc31","Type":"ContainerStarted","Data":"93aba789ec1fc723aa7a2ce6661f26eb322c0822e182df3771cd071c5b3b3b24"} Jul 15 13:54:32.050087 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:32.050033 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" podStartSLOduration=2.050019216 podStartE2EDuration="2.050019216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:54:32.048800043 +0000 UTC m=+2878.004254610" watchObservedRunningTime="2026-07-15 13:54:32.050019216 +0000 UTC m=+2878.005473779" Jul 15 13:54:41.113198 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:41.113163 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:41.113656 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:41.113213 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:41.125620 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:41.125598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:54:42.094073 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:54:42.094041 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:55:14.619918 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:14.619886 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn"] Jul 15 13:55:29.640401 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.640334 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" podUID="d6ce0213-c081-4615-98ed-0ab2794bbc31" containerName="main" containerID="cri-o://31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25" gracePeriod=45 Jul 15 13:55:29.898977 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.898902 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:55:29.983484 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983441 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-tmp-dir\") pod \"d6ce0213-c081-4615-98ed-0ab2794bbc31\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " Jul 15 13:55:29.983672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983530 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8s89\" (UniqueName: \"kubernetes.io/projected/d6ce0213-c081-4615-98ed-0ab2794bbc31-kube-api-access-c8s89\") pod \"d6ce0213-c081-4615-98ed-0ab2794bbc31\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " Jul 15 13:55:29.983672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983565 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ce0213-c081-4615-98ed-0ab2794bbc31-tls-certs\") pod \"d6ce0213-c081-4615-98ed-0ab2794bbc31\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " Jul 15 13:55:29.983672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983593 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-home\") pod \"d6ce0213-c081-4615-98ed-0ab2794bbc31\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " Jul 15 13:55:29.983672 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983668 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-model-cache\") pod \"d6ce0213-c081-4615-98ed-0ab2794bbc31\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " Jul 15 13:55:29.983883 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983690 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-dshm\") pod \"d6ce0213-c081-4615-98ed-0ab2794bbc31\" (UID: \"d6ce0213-c081-4615-98ed-0ab2794bbc31\") " Jul 15 13:55:29.983883 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983763 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d6ce0213-c081-4615-98ed-0ab2794bbc31" (UID: "d6ce0213-c081-4615-98ed-0ab2794bbc31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.983883 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983828 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-home" (OuterVolumeSpecName: "home") pod "d6ce0213-c081-4615-98ed-0ab2794bbc31" (UID: "d6ce0213-c081-4615-98ed-0ab2794bbc31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.984039 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.983941 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-model-cache" (OuterVolumeSpecName: "model-cache") pod "d6ce0213-c081-4615-98ed-0ab2794bbc31" (UID: "d6ce0213-c081-4615-98ed-0ab2794bbc31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:29.984039 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.984006 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-tmp-dir\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.984039 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.984027 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-home\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:55:29.985758 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.985732 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6ce0213-c081-4615-98ed-0ab2794bbc31-kube-api-access-c8s89" (OuterVolumeSpecName: "kube-api-access-c8s89") pod "d6ce0213-c081-4615-98ed-0ab2794bbc31" (UID: "d6ce0213-c081-4615-98ed-0ab2794bbc31"). InnerVolumeSpecName "kube-api-access-c8s89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:55:29.985758 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.985746 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6ce0213-c081-4615-98ed-0ab2794bbc31-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d6ce0213-c081-4615-98ed-0ab2794bbc31" (UID: "d6ce0213-c081-4615-98ed-0ab2794bbc31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 15 13:55:29.985897 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:29.985790 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-dshm" (OuterVolumeSpecName: "dshm") pod "d6ce0213-c081-4615-98ed-0ab2794bbc31" (UID: "d6ce0213-c081-4615-98ed-0ab2794bbc31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:55:30.085075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.085041 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8s89\" (UniqueName: \"kubernetes.io/projected/d6ce0213-c081-4615-98ed-0ab2794bbc31-kube-api-access-c8s89\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:55:30.085075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.085068 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6ce0213-c081-4615-98ed-0ab2794bbc31-tls-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:55:30.085075 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.085077 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-model-cache\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:55:30.085295 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.085086 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6ce0213-c081-4615-98ed-0ab2794bbc31-dshm\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:55:30.337862 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.337829 2564 generic.go:356] "Generic (PLEG): container finished" podID="d6ce0213-c081-4615-98ed-0ab2794bbc31" containerID="31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25" exitCode=0 Jul 15 13:55:30.338035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.337874 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" Jul 15 13:55:30.338035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.337922 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" event={"ID":"d6ce0213-c081-4615-98ed-0ab2794bbc31","Type":"ContainerDied","Data":"31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25"} Jul 15 13:55:30.338035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.337968 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn" event={"ID":"d6ce0213-c081-4615-98ed-0ab2794bbc31","Type":"ContainerDied","Data":"93aba789ec1fc723aa7a2ce6661f26eb322c0822e182df3771cd071c5b3b3b24"} Jul 15 13:55:30.338035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.337985 2564 scope.go:117] "RemoveContainer" containerID="31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25" Jul 15 13:55:30.349868 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.349852 2564 scope.go:117] "RemoveContainer" containerID="31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25" Jul 15 13:55:30.350115 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:55:30.350098 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25\": container with ID starting with 31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25 not found: ID does not exist" containerID="31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25" Jul 15 13:55:30.350172 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.350122 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25"} err="failed to get container status \"31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25\": rpc error: code = NotFound desc = could not find container \"31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25\": container with ID starting with 31145d2e9ebad87283b9c058f415fe4d2d308ac810211ae04c4672d6092a5d25 not found: ID does not exist" Jul 15 13:55:30.362607 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.362583 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn"] Jul 15 13:55:30.366854 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.366834 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-bc6fa6/rate-test-model-bc6fa6-kserve-7fc8bf6f47-b7fjn"] Jul 15 13:55:30.664580 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:30.664495 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6ce0213-c081-4615-98ed-0ab2794bbc31" path="/var/lib/kubelet/pods/d6ce0213-c081-4615-98ed-0ab2794bbc31/volumes" Jul 15 13:55:59.409147 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:59.409070 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j"] Jul 15 13:55:59.409651 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:55:59.409293 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" containerID="cri-o://955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75" gracePeriod=30 Jul 15 13:56:00.298035 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:00.298003 2564 patch_prober.go:28] interesting pod/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 15 13:56:00.298214 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:00.298071 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 15 13:56:04.665136 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.665112 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:56:04.801390 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-podinfo\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801407 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-data\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801423 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-certs\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801486 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4m45\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-kube-api-access-w4m45\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801515 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-token\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801563 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801551 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08af9f30-decd-4e95-8229-cf61cddfee0e-istiod-ca-cert\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801815 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801572 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-credential-socket\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801815 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801639 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-envoy\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801815 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801681 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-socket\") pod \"08af9f30-decd-4e95-8229-cf61cddfee0e\" (UID: \"08af9f30-decd-4e95-8229-cf61cddfee0e\") " Jul 15 13:56:04.801976 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801938 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:04.802038 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.801983 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08af9f30-decd-4e95-8229-cf61cddfee0e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:56:04.802247 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.802199 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:04.802375 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.802254 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:04.802375 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.802306 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-data" (OuterVolumeSpecName: "istio-data") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:04.803794 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.803761 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-kube-api-access-w4m45" (OuterVolumeSpecName: "kube-api-access-w4m45") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "kube-api-access-w4m45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:56:04.803794 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.803759 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-token" (OuterVolumeSpecName: "istio-token") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:56:04.803936 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.803804 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:56:04.803936 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.803886 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "08af9f30-decd-4e95-8229-cf61cddfee0e" (UID: "08af9f30-decd-4e95-8229-cf61cddfee0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:04.903121 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903095 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903121 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903118 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/08af9f30-decd-4e95-8229-cf61cddfee0e-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903128 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903137 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903146 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903154 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903161 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903169 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/08af9f30-decd-4e95-8229-cf61cddfee0e-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:04.903284 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:04.903177 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4m45\" (UniqueName: \"kubernetes.io/projected/08af9f30-decd-4e95-8229-cf61cddfee0e-kube-api-access-w4m45\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:05.523675 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.523643 2564 generic.go:356] "Generic (PLEG): container finished" podID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerID="955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75" exitCode=0 Jul 15 13:56:05.523833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.523685 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" event={"ID":"08af9f30-decd-4e95-8229-cf61cddfee0e","Type":"ContainerDied","Data":"955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75"} Jul 15 13:56:05.523833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.523689 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" Jul 15 13:56:05.523833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.523728 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j" event={"ID":"08af9f30-decd-4e95-8229-cf61cddfee0e","Type":"ContainerDied","Data":"68d47b5648043287b0789737aa0a75d2d4e411eb78ece3f6e4050309fc3341cf"} Jul 15 13:56:05.523833 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.523749 2564 scope.go:117] "RemoveContainer" containerID="955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75" Jul 15 13:56:05.533307 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.533138 2564 scope.go:117] "RemoveContainer" containerID="57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119" Jul 15 13:56:05.541209 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.541191 2564 scope.go:117] "RemoveContainer" containerID="955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75" Jul 15 13:56:05.541441 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:56:05.541418 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75\": container with ID starting with 955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75 not found: ID does not exist" containerID="955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75" Jul 15 13:56:05.541527 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.541451 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75"} err="failed to get container status \"955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75\": rpc error: code = NotFound desc = could not find container \"955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75\": container with ID starting with 955e39b2d448cb059c833c3c7dce5f023585b3cb8ec8820f925f69286c42da75 not found: ID does not exist" Jul 15 13:56:05.541527 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.541490 2564 scope.go:117] "RemoveContainer" containerID="57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119" Jul 15 13:56:05.541735 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:56:05.541720 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119\": container with ID starting with 57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119 not found: ID does not exist" containerID="57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119" Jul 15 13:56:05.541782 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.541739 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119"} err="failed to get container status \"57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119\": rpc error: code = NotFound desc = could not find container \"57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119\": container with ID starting with 57aaf6b47509070587a77f1963b83bedacf70e78c119052eafd45500eb4cc119 not found: ID does not exist" Jul 15 13:56:05.546415 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.546395 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j"] Jul 15 13:56:05.550283 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:05.550265 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-7abf62-openshift-default-857f77f59-q9h5j"] Jul 15 13:56:06.656714 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:06.656687 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" path="/var/lib/kubelet/pods/08af9f30-decd-4e95-8229-cf61cddfee0e/volumes" Jul 15 13:56:25.648026 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:25.647992 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg"] Jul 15 13:56:25.648485 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:25.648259 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" containerID="cri-o://4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb" gracePeriod=30 Jul 15 13:56:26.386544 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:26.386514 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-856d6647d4-gxvs2_8816b4bc-ad61-470d-acc4-1218690a4730/authorino/0.log" Jul 15 13:56:30.907379 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:30.907357 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:56:31.036716 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-token\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.036888 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036728 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-certs\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.036888 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036760 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-envoy\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.036888 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-data\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.036888 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036822 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istiod-ca-cert\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.036888 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036847 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-credential-socket\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.036888 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036885 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-socket\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.037257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036920 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-podinfo\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.037257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.036960 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq8kr\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-kube-api-access-fq8kr\") pod \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\" (UID: \"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8\") " Jul 15 13:56:31.037257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:31.037257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037172 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:31.037257 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037203 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 15 13:56:31.037537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037323 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:31.037589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037548 2564 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-certs\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.037589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037570 2564 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istiod-ca-cert\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.037589 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037583 2564 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-credential-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.037742 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037599 2564 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-workload-socket\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.037828 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.037805 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:31.038960 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.038939 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:56:31.039072 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.039030 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 15 13:56:31.039545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.039520 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 15 13:56:31.039545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.039533 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-kube-api-access-fq8kr" (OuterVolumeSpecName: "kube-api-access-fq8kr") pod "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" (UID: "6363c76f-ff3e-4e9e-8951-e9e8d1232dd8"). InnerVolumeSpecName "kube-api-access-fq8kr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 15 13:56:31.138723 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.138675 2564 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-podinfo\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.138723 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.138717 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq8kr\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-kube-api-access-fq8kr\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.138723 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.138732 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-token\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.138978 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.138745 2564 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-envoy\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.138978 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.138757 2564 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8-istio-data\") on node \"ip-10-0-139-149.ec2.internal\" DevicePath \"\"" Jul 15 13:56:31.672331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.672297 2564 generic.go:356] "Generic (PLEG): container finished" podID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerID="4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb" exitCode=0 Jul 15 13:56:31.672505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.672347 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" Jul 15 13:56:31.672505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.672388 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" event={"ID":"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8","Type":"ContainerDied","Data":"4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb"} Jul 15 13:56:31.672505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.672442 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg" event={"ID":"6363c76f-ff3e-4e9e-8951-e9e8d1232dd8","Type":"ContainerDied","Data":"8b3b4cd68649e4d2aaac9debcd65d99c82e4b449e9dd0372d66b16c4afe61b5d"} Jul 15 13:56:31.672505 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.672479 2564 scope.go:117] "RemoveContainer" containerID="4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb" Jul 15 13:56:31.682339 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.682322 2564 scope.go:117] "RemoveContainer" containerID="d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927" Jul 15 13:56:31.690068 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.690053 2564 scope.go:117] "RemoveContainer" containerID="4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb" Jul 15 13:56:31.690289 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:56:31.690273 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb\": container with ID starting with 4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb not found: ID does not exist" containerID="4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb" Jul 15 13:56:31.690338 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.690299 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb"} err="failed to get container status \"4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb\": rpc error: code = NotFound desc = could not find container \"4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb\": container with ID starting with 4db1985e09694ce1458d8a13f19d55ce32d0faedee6e5c591da3103acac5fdfb not found: ID does not exist" Jul 15 13:56:31.690338 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.690320 2564 scope.go:117] "RemoveContainer" containerID="d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927" Jul 15 13:56:31.690551 ip-10-0-139-149 kubenswrapper[2564]: E0715 13:56:31.690534 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927\": container with ID starting with d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927 not found: ID does not exist" containerID="d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927" Jul 15 13:56:31.690590 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.690558 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927"} err="failed to get container status \"d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927\": rpc error: code = NotFound desc = could not find container \"d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927\": container with ID starting with d4e78660a76f8079ccb11e716f4c4a87c4680576964f31b8d7fecb0dea444927 not found: ID does not exist" Jul 15 13:56:31.694000 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.693980 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg"] Jul 15 13:56:31.700046 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:31.700027 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-bc6fa6-openshift-default-558fb8f5bc-2bpzg"] Jul 15 13:56:32.657319 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:32.657289 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" path="/var/lib/kubelet/pods/6363c76f-ff3e-4e9e-8951-e9e8d1232dd8/volumes" Jul 15 13:56:32.868543 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:32.868518 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-rfq5q_8c24eee3-8564-4b2c-8c87-a59c70fc1da4/copy-manifests/0.log" Jul 15 13:56:32.880613 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:32.880592 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-rfq5q_8c24eee3-8564-4b2c-8c87-a59c70fc1da4/manager/0.log" Jul 15 13:56:33.369453 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:33.369424 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735370-xx8c4_72ffe8c5-6286-4662-b8c4-11a1aee1a704/cleanup/0.log" Jul 15 13:56:33.488032 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:33.488007 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29735385-k2wb4_83db9a48-aee8-4d05-8a0a-094740b6b5ac/cleanup/0.log" Jul 15 13:56:33.603010 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:33.602989 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-b9daa40c-mv89m_d2767e48-be80-4a67-b613-9b189ed13d9c/revoke-keys/0.log" Jul 15 13:56:33.718739 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:33.718668 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-fb3eae66-cf2kw_ccc5f0da-44e8-4ba2-adb9-e9dbaee4fcea/revoke-keys/0.log" Jul 15 13:56:34.073300 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:34.073273 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-c15b0ffb-v7lsc_1c56b591-8cd7-4b08-add3-6348acf42b77/revoke-keys/0.log" Jul 15 13:56:34.416830 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:34.416758 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-faa562cb-5q9vs_5a22eacc-89bb-427b-aba2-68e9bf1d8f9c/revoke-keys/0.log" Jul 15 13:56:34.529912 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:34.529879 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-b14f8bd8-kwbqq_62852d60-3940-4069-a66c-dde10bd6d94b/revoke-keys/0.log" Jul 15 13:56:34.806662 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:34.806634 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:56:34.820159 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:34.820136 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-ghhfs_2c444ed9-604e-4fae-8437-7b0548983837/ovn-acl-logging/0.log" Jul 15 13:56:36.302029 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:36.302000 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-856d6647d4-gxvs2_8816b4bc-ad61-470d-acc4-1218690a4730/authorino/0.log" Jul 15 13:56:36.424614 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:36.424582 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-f2tcp_c8d90403-666b-4c64-9ebe-7ff8419c4b44/manager/0.log" Jul 15 13:56:37.903068 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:37.903041 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-5k7bw_799fe46c-9193-4755-858d-0d60152b4fc1/kube-auth-proxy/0.log" Jul 15 13:56:38.142767 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:38.142739 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-jrrl8_5b80e131-71e9-41ed-81cf-d6530500f3bc/payload-pre-processing/0.log" Jul 15 13:56:38.383689 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:38.383639 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dd8dbb64-8vkwx_8dbe8e4b-d986-49e1-a374-01d806777597/router/0.log" Jul 15 13:56:38.732868 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:38.732788 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-mxn75_83774881-a7a0-4cf9-834e-bd003a90ed29/main/0.log" Jul 15 13:56:38.963845 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:38.963816 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-x98xx_fba6581a-79aa-4320-ab58-9df4b8a113d3/main/0.log" Jul 15 13:56:45.141063 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141035 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mvq2f/must-gather-gp8mg"] Jul 15 13:56:45.141490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141417 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6ce0213-c081-4615-98ed-0ab2794bbc31" containerName="main" Jul 15 13:56:45.141490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141432 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6ce0213-c081-4615-98ed-0ab2794bbc31" containerName="main" Jul 15 13:56:45.141490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141449 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" Jul 15 13:56:45.141490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141458 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" Jul 15 13:56:45.141490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141484 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" Jul 15 13:56:45.141490 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141493 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141506 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141515 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141624 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141639 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6363c76f-ff3e-4e9e-8951-e9e8d1232dd8" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141649 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6ce0213-c081-4615-98ed-0ab2794bbc31" containerName="main" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141659 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141757 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" Jul 15 13:56:45.141824 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141767 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" Jul 15 13:56:45.142100 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.141862 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="08af9f30-decd-4e95-8229-cf61cddfee0e" containerName="istio-proxy" Jul 15 13:56:45.144691 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.144677 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.147124 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.147103 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-mvq2f\"/\"default-dockercfg-wct6h\"" Jul 15 13:56:45.147398 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.147377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mvq2f\"/\"kube-root-ca.crt\"" Jul 15 13:56:45.147491 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.147453 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-mvq2f\"/\"openshift-service-ca.crt\"" Jul 15 13:56:45.153260 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.153239 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mvq2f/must-gather-gp8mg"] Jul 15 13:56:45.266539 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.266508 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7bg7\" (UniqueName: \"kubernetes.io/projected/6cf90734-eff0-4a0b-9c60-0c4b29bea0bd-kube-api-access-n7bg7\") pod \"must-gather-gp8mg\" (UID: \"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd\") " pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.266683 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.266555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf90734-eff0-4a0b-9c60-0c4b29bea0bd-must-gather-output\") pod \"must-gather-gp8mg\" (UID: \"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd\") " pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.367155 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.367124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7bg7\" (UniqueName: \"kubernetes.io/projected/6cf90734-eff0-4a0b-9c60-0c4b29bea0bd-kube-api-access-n7bg7\") pod \"must-gather-gp8mg\" (UID: \"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd\") " pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.367308 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.367164 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf90734-eff0-4a0b-9c60-0c4b29bea0bd-must-gather-output\") pod \"must-gather-gp8mg\" (UID: \"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd\") " pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.367537 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.367515 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6cf90734-eff0-4a0b-9c60-0c4b29bea0bd-must-gather-output\") pod \"must-gather-gp8mg\" (UID: \"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd\") " pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.376000 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.375978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7bg7\" (UniqueName: \"kubernetes.io/projected/6cf90734-eff0-4a0b-9c60-0c4b29bea0bd-kube-api-access-n7bg7\") pod \"must-gather-gp8mg\" (UID: \"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd\") " pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.454529 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.454458 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mvq2f/must-gather-gp8mg" Jul 15 13:56:45.583942 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.583906 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mvq2f/must-gather-gp8mg"] Jul 15 13:56:45.587225 ip-10-0-139-149 kubenswrapper[2564]: W0715 13:56:45.587198 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cf90734_eff0_4a0b_9c60_0c4b29bea0bd.slice/crio-44c5c35adb954e1b177e9e39d91fd13f4eb9a9794cbf505a37ab23f4abfea192 WatchSource:0}: Error finding container 44c5c35adb954e1b177e9e39d91fd13f4eb9a9794cbf505a37ab23f4abfea192: Status 404 returned error can't find the container with id 44c5c35adb954e1b177e9e39d91fd13f4eb9a9794cbf505a37ab23f4abfea192 Jul 15 13:56:45.589275 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.589255 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 15 13:56:45.745056 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:45.744992 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/must-gather-gp8mg" event={"ID":"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd","Type":"ContainerStarted","Data":"44c5c35adb954e1b177e9e39d91fd13f4eb9a9794cbf505a37ab23f4abfea192"} Jul 15 13:56:46.752817 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:46.752779 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/must-gather-gp8mg" event={"ID":"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd","Type":"ContainerStarted","Data":"8031198bd4847526e358de0c7ee3244249bb61a3bc991313852962dd8e68e257"} Jul 15 13:56:46.753289 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:46.752838 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/must-gather-gp8mg" event={"ID":"6cf90734-eff0-4a0b-9c60-0c4b29bea0bd","Type":"ContainerStarted","Data":"00c8cb7dace23735d554611876bc042091394147c9d3fedae84e550bfe72dcc8"} Jul 15 13:56:46.768460 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:46.768400 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mvq2f/must-gather-gp8mg" podStartSLOduration=1.088505592 podStartE2EDuration="1.768377265s" totalImagesPullingTime="679.871673ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-15 13:56:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:56:46.76690794 +0000 UTC m=+3012.722362527" watchObservedRunningTime="2026-07-15 13:56:46.768377265 +0000 UTC m=+3012.723831831" Jul 15 13:56:47.900283 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:47.900226 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-lgnvn_18e85af3-fa0a-40e5-9350-2133b1797824/global-pull-secret-syncer/0.log" Jul 15 13:56:48.026795 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:48.026768 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jgwwr_33145dcd-966e-41d4-b9e7-b0c1dfe16528/konnectivity-agent/0.log" Jul 15 13:56:48.104110 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:48.104079 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-149.ec2.internal_0dd7a4f18fc1aca27c6a9caeb4dafb15/haproxy/0.log" Jul 15 13:56:52.764743 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:52.764702 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-856d6647d4-gxvs2_8816b4bc-ad61-470d-acc4-1218690a4730/authorino/0.log" Jul 15 13:56:52.805370 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:52.805103 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-f2tcp_c8d90403-666b-4c64-9ebe-7ff8419c4b44/manager/0.log" Jul 15 13:56:54.720101 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.720072 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/alertmanager/0.log" Jul 15 13:56:54.742747 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.742710 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/config-reloader/0.log" Jul 15 13:56:54.764740 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.764705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/kube-rbac-proxy-web/0.log" Jul 15 13:56:54.787769 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.787726 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/kube-rbac-proxy/0.log" Jul 15 13:56:54.816007 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.815934 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/kube-rbac-proxy-metric/0.log" Jul 15 13:56:54.837369 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.837339 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/prom-label-proxy/0.log" Jul 15 13:56:54.859222 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:54.859170 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d5252322-2e18-42fb-b72b-392ef40ceacc/init-config-reloader/0.log" Jul 15 13:56:55.013806 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.013687 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-glk2f_fcf69ac6-c3bb-4579-94f7-93d18204abb2/kube-state-metrics/0.log" Jul 15 13:56:55.030937 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.030798 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-glk2f_fcf69ac6-c3bb-4579-94f7-93d18204abb2/kube-rbac-proxy-main/0.log" Jul 15 13:56:55.051800 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.051666 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-785778ff74-glk2f_fcf69ac6-c3bb-4579-94f7-93d18204abb2/kube-rbac-proxy-self/0.log" Jul 15 13:56:55.080737 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.080648 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-78999c7f87-qbnww_1b9ff85e-f5c1-4fb5-919d-d3e7c18d9970/metrics-server/0.log" Jul 15 13:56:55.224103 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.224067 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2cwm_f2aeca5e-acaf-415c-aa19-44ebf58cbd4d/node-exporter/0.log" Jul 15 13:56:55.244736 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.244705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2cwm_f2aeca5e-acaf-415c-aa19-44ebf58cbd4d/kube-rbac-proxy/0.log" Jul 15 13:56:55.266956 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.266866 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-c2cwm_f2aeca5e-acaf-415c-aa19-44ebf58cbd4d/init-textfile/0.log" Jul 15 13:56:55.379545 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.379421 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6qg55_25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba/kube-rbac-proxy-main/0.log" Jul 15 13:56:55.406036 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.405995 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6qg55_25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba/kube-rbac-proxy-self/0.log" Jul 15 13:56:55.443676 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.443637 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-84bd68b896-6qg55_25ca13b1-d6d0-44e6-b8a7-89e0a1e46aba/openshift-state-metrics/0.log" Jul 15 13:56:55.503657 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.503631 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/prometheus/0.log" Jul 15 13:56:55.536532 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.536506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/config-reloader/0.log" Jul 15 13:56:55.576797 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.576771 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/thanos-sidecar/0.log" Jul 15 13:56:55.622498 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.622411 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/kube-rbac-proxy-web/0.log" Jul 15 13:56:55.661668 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.661566 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/kube-rbac-proxy/0.log" Jul 15 13:56:55.698868 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.698831 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/kube-rbac-proxy-thanos/0.log" Jul 15 13:56:55.738199 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.738174 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e57f676a-b7e9-4617-b5da-7ddf415925c1/init-config-reloader/0.log" Jul 15 13:56:55.901435 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.901366 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-glr7n_a032e11a-e613-45d0-9cfd-db25e81633e7/prometheus-operator-admission-webhook/0.log" Jul 15 13:56:55.944264 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.944193 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c687b8b78-svdjj_fcec410d-4f20-4660-a555-80d334ef15c7/telemeter-client/0.log" Jul 15 13:56:55.978562 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:55.978517 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c687b8b78-svdjj_fcec410d-4f20-4660-a555-80d334ef15c7/reload/0.log" Jul 15 13:56:56.014614 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.014582 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c687b8b78-svdjj_fcec410d-4f20-4660-a555-80d334ef15c7/kube-rbac-proxy/0.log" Jul 15 13:56:56.059332 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.059297 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6897b47b98-8v9z9_f6aec2de-f43e-4b1f-aed1-fc326495ce6e/thanos-query/0.log" Jul 15 13:56:56.097693 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.097660 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6897b47b98-8v9z9_f6aec2de-f43e-4b1f-aed1-fc326495ce6e/kube-rbac-proxy-web/0.log" Jul 15 13:56:56.134140 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.134108 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6897b47b98-8v9z9_f6aec2de-f43e-4b1f-aed1-fc326495ce6e/kube-rbac-proxy/0.log" Jul 15 13:56:56.173169 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.173136 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6897b47b98-8v9z9_f6aec2de-f43e-4b1f-aed1-fc326495ce6e/prom-label-proxy/0.log" Jul 15 13:56:56.208428 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.208340 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6897b47b98-8v9z9_f6aec2de-f43e-4b1f-aed1-fc326495ce6e/kube-rbac-proxy-rules/0.log" Jul 15 13:56:56.250741 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:56.250702 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6897b47b98-8v9z9_f6aec2de-f43e-4b1f-aed1-fc326495ce6e/kube-rbac-proxy-metrics/0.log" Jul 15 13:56:58.308250 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.308217 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm"] Jul 15 13:56:58.314098 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.314066 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.328432 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.328403 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm"] Jul 15 13:56:58.413550 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.413507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-podres\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.413744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.413595 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-lib-modules\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.413744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.413633 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-sys\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.413744 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.413732 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkjd5\" (UniqueName: \"kubernetes.io/projected/aa9274ea-bad4-4854-924a-3943d96b9bcd-kube-api-access-lkjd5\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.414040 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.413790 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-proc\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515374 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515330 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-lib-modules\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-sys\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkjd5\" (UniqueName: \"kubernetes.io/projected/aa9274ea-bad4-4854-924a-3943d96b9bcd-kube-api-access-lkjd5\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-proc\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515595 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515591 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-podres\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515823 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515759 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-podres\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515878 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515860 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-lib-modules\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.515935 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.515905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-sys\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.516276 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.516251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/aa9274ea-bad4-4854-924a-3943d96b9bcd-proc\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.524170 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.524143 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkjd5\" (UniqueName: \"kubernetes.io/projected/aa9274ea-bad4-4854-924a-3943d96b9bcd-kube-api-access-lkjd5\") pod \"perf-node-gather-daemonset-4hnwm\" (UID: \"aa9274ea-bad4-4854-924a-3943d96b9bcd\") " pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.635237 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.635158 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:58.677109 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.677082 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-59646cffc8-tk9hx_4d89908d-52a0-4b9c-af0a-e91980b349e4/console/0.log" Jul 15 13:56:58.812080 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.812043 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm"] Jul 15 13:56:58.863715 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:58.861976 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" event={"ID":"aa9274ea-bad4-4854-924a-3943d96b9bcd","Type":"ContainerStarted","Data":"1e5dcd3c1e6422b1c556e7036264e562362def5d3d5dad09a27397b282d1e5a6"} Jul 15 13:56:59.357338 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:59.357305 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-84dnb_dd675c67-109e-455e-a40b-97166836645a/volume-data-source-validator/0.log" Jul 15 13:56:59.871254 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:59.871218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" event={"ID":"aa9274ea-bad4-4854-924a-3943d96b9bcd","Type":"ContainerStarted","Data":"8f102c1c03b1963d79d7ae378c83093c337069f18ece506a4f7ff6299b1b0f83"} Jul 15 13:56:59.871680 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:59.871646 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:56:59.890324 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:56:59.890268 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" podStartSLOduration=1.890235646 podStartE2EDuration="1.890235646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-15 13:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-15 13:56:59.889182468 +0000 UTC m=+3025.844637031" watchObservedRunningTime="2026-07-15 13:56:59.890235646 +0000 UTC m=+3025.845690212" Jul 15 13:57:00.505297 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:00.505270 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wvk9l_65a06378-bbbd-412e-95fe-21d586ec36ea/dns/0.log" Jul 15 13:57:00.526710 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:00.526683 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wvk9l_65a06378-bbbd-412e-95fe-21d586ec36ea/kube-rbac-proxy/0.log" Jul 15 13:57:00.553908 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:00.553880 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-c4wgv_c5b39e97-5dd3-4472-b87f-436050a3656e/dns-node-resolver/0.log" Jul 15 13:57:01.217766 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:01.217728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-x7jkc_03957cc5-0ec0-48c6-a682-ab1a0d26a273/node-ca/0.log" Jul 15 13:57:02.302938 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:02.302894 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-5k7bw_799fe46c-9193-4755-858d-0d60152b4fc1/kube-auth-proxy/0.log" Jul 15 13:57:02.393331 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:02.393300 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-jrrl8_5b80e131-71e9-41ed-81cf-d6530500f3bc/payload-pre-processing/0.log" Jul 15 13:57:02.454031 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:02.454004 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dd8dbb64-8vkwx_8dbe8e4b-d986-49e1-a374-01d806777597/router/0.log" Jul 15 13:57:03.192543 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:03.192506 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kgndk_0030efc9-4bad-477c-9a8c-4ba925b0e420/serve-healthcheck-canary/0.log" Jul 15 13:57:03.700022 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:03.699991 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hqsl_499499f3-41c2-4699-98d0-69c986e2faf3/kube-rbac-proxy/0.log" Jul 15 13:57:03.720040 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:03.720013 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hqsl_499499f3-41c2-4699-98d0-69c986e2faf3/exporter/0.log" Jul 15 13:57:03.740709 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:03.740681 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8hqsl_499499f3-41c2-4699-98d0-69c986e2faf3/extractor/0.log" Jul 15 13:57:05.889799 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:05.889771 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-mvq2f/perf-node-gather-daemonset-4hnwm" Jul 15 13:57:06.512647 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:06.512608 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-rfq5q_8c24eee3-8564-4b2c-8c87-a59c70fc1da4/manager/0.log" Jul 15 13:57:06.533056 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:06.532976 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-869c48857d-rfq5q_8c24eee3-8564-4b2c-8c87-a59c70fc1da4/copy-manifests/0.log" Jul 15 13:57:07.903522 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:07.903491 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8ddc6f8f9-l7q87_0ae2fb56-f03c-48c4-8943-62cfb5dad3ec/manager/0.log" Jul 15 13:57:08.601668 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:08.601633 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-5k7bw_799fe46c-9193-4755-858d-0d60152b4fc1/kube-auth-proxy/0.log" Jul 15 13:57:08.667364 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:08.667335 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-jrrl8_5b80e131-71e9-41ed-81cf-d6530500f3bc/payload-pre-processing/0.log" Jul 15 13:57:08.696538 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:08.696508 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dd8dbb64-8vkwx_8dbe8e4b-d986-49e1-a374-01d806777597/router/0.log" Jul 15 13:57:10.005688 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:10.005624 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b6cfc997-5k7bw_799fe46c-9193-4755-858d-0d60152b4fc1/kube-auth-proxy/0.log" Jul 15 13:57:10.078805 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:10.078774 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-74584bb9f8-jrrl8_5b80e131-71e9-41ed-81cf-d6530500f3bc/payload-pre-processing/0.log" Jul 15 13:57:10.116067 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:10.116031 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66dd8dbb64-8vkwx_8dbe8e4b-d986-49e1-a374-01d806777597/router/0.log" Jul 15 13:57:13.382820 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:13.382791 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-2fjtw_c783ddea-55cf-4263-8db7-d8d43cea99f2/kube-storage-version-migrator-operator/1.log" Jul 15 13:57:13.383647 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:13.383628 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-2fjtw_c783ddea-55cf-4263-8db7-d8d43cea99f2/kube-storage-version-migrator-operator/0.log" Jul 15 13:57:14.725943 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.725913 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/kube-multus-additional-cni-plugins/0.log" Jul 15 13:57:14.745753 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.745728 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/egress-router-binary-copy/0.log" Jul 15 13:57:14.766173 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.766148 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/cni-plugins/0.log" Jul 15 13:57:14.787690 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.787669 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/bond-cni-plugin/0.log" Jul 15 13:57:14.806727 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.806705 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/routeoverride-cni/0.log" Jul 15 13:57:14.826074 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.826054 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/whereabouts-cni-bincopy/0.log" Jul 15 13:57:14.845599 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.845571 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-spqmk_6349c6b9-5951-4fa4-97b5-6d0497b68807/whereabouts-cni/0.log" Jul 15 13:57:14.941977 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:14.941947 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-nwkk4_0d4460e2-5dfa-42ab-9921-6a617c2d207d/kube-multus/0.log" Jul 15 13:57:15.005106 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:15.005031 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5vjbj_dea320b5-7fcb-48e0-bf9e-b9475cfb141a/network-metrics-daemon/0.log" Jul 15 13:57:15.022619 ip-10-0-139-149 kubenswrapper[2564]: I0715 13:57:15.022591 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-5vjbj_dea320b5-7fcb-48e0-bf9e-b9475cfb141a/kube-rbac-proxy/0.log"